Re: [saag] Algorithms/modes requested by users/customers

Paul Hoffman <paul.hoffman@vpnc.org> Tue, 19 February 2008 16:19 UTC

Received: from pacific-carrier-annex.mit.edu (PACIFIC-CARRIER-ANNEX.MIT.EDU [18.7.21.83]) by pch.mit.edu (8.13.6/8.12.8) with ESMTP id m1JGJIdo001231 for <saag@PCH.mit.edu>; Tue, 19 Feb 2008 11:19:18 -0500
Received: from mit.edu (W92-130-BARRACUDA-2.MIT.EDU [18.7.21.223]) by pacific-carrier-annex.mit.edu (8.13.6/8.9.2) with ESMTP id m1JGJAd0019344 for <saag@mit.edu>; Tue, 19 Feb 2008 11:19:11 -0500 (EST)
Received: from balder-227.proper.com (Balder-227.Proper.COM [192.245.12.227]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mit.edu (Spam Firewall) with ESMTP id 76AE5C9E700 for <saag@mit.edu>; Tue, 19 Feb 2008 11:18:49 -0500 (EST)
Received: from [10.20.30.152] (dsl-63-249-108-169.cruzio.com [63.249.108.169]) (authenticated bits=0) by balder-227.proper.com (8.13.5/8.13.5) with ESMTP id m1JGIQhh073200 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 19 Feb 2008 09:18:28 -0700 (MST) (envelope-from paul.hoffman@vpnc.org)
Mime-Version: 1.0
Message-Id: <p06240804c3e0ad5d1fa4@[10.20.30.152]>
In-Reply-To: <p06240504c3e09559649c@[192.168.0.102]>
References: <8329C86009B2F24493D76B486146769A9429B7A8@USEXCHANGE.corp.extremenetworks . com> <p06240804c3de211f0592@[10.20.30.162]> <p06240504c3e09559649c@[192.168.0.102]>
Date: Tue, 19 Feb 2008 08:18:24 -0800
To: Stephen Kent <kent@bbn.com>
From: Paul Hoffman <paul.hoffman@vpnc.org>
Content-Type: text/plain; charset="us-ascii"; format="flowed"
X-Spam-Score: 0.00
X-Spam-Flag: NO
X-Scanned-By: MIMEDefang 2.42
Cc: "saag@mit.edu" <saag@mit.edu>, Randall Atkinson <rja@extremenetworks.com>
Subject: Re: [saag] Algorithms/modes requested by users/customers
X-BeenThere: saag@mit.edu
X-Mailman-Version: 2.1.6
Precedence: list
List-Id: IETF Security Area Advisory Group <saag.mit.edu>
List-Unsubscribe: <http://mailman.mit.edu/mailman/listinfo/saag>, <mailto:saag-request@mit.edu?subject=unsubscribe>
List-Archive: <http://mailman.mit.edu/pipermail/saag>
List-Post: <mailto:saag@mit.edu>
List-Help: <mailto:saag-request@mit.edu?subject=help>
List-Subscribe: <http://mailman.mit.edu/mailman/listinfo/saag>, <mailto:saag-request@mit.edu?subject=subscribe>
X-List-Received-Date: Tue, 19 Feb 2008 16:19:18 -0000

At 10:15 AM -0500 2/19/08, Stephen Kent wrote:
>Can you share the reasons cited by vendors to support the notion 
>that the FIPS 140 process is broken.

Sure.

- It takes way too long from submission of system to validation, even 
if no problems are found.

- If problems are found during evaluation, the restart time is too long.

- Some of the tests are fairly subjective, and it becomes a game of 
fixing code to please the testing service, not to make the product 
more secure.

- An evaluated product cannot have its core firmware updated without 
losing the validation. For example, if a customer asks for a new 
feature that might touch the crypto, the vendor has to choose between 
losing the validation (and paying for a new one, and waiting for the 
results) or keeping the customer happy.

- The validation doesn't even check for on-the-wire interoperability, 
which is what the customers care about most.

- The test process is too expensive for many low-end devices that 
would be very useful in USGovt offices.

- The system introduces silly modes that make the systems more complicated.

These are the top complaints I hear from both large and small 
vendors. There are certainly others. "Oh, don't get me started" is 
commonly heard when discussing FIPS 140 testing.

>   Compared to other security evaluation criteria, e.g. CC or the old 
>Organge Book, most security folks I know view the FIPS 140 
>evaluation program as well managed and not very onerous.

"Security folks" are not good evaluators on how some processes affect 
the market.

>Also, if buyers believe that a device that "could be evaluated" is 
>good enough, they are being rather naive.

Maybe. If a buyer cares about "does this box support the needed 
crypto in an interoperable fashion", then the perception is fine. If 
they care about all the details that FIPS 140 testing covers, then of 
course it is naive.

>Experience with the FIPS 140 program has shown that a significant 
>fraction of of products submitted for evaluation fail the process.

This statement is usually bandied about without any quantification of 
what failures were found, as it is here. As someone who creates test 
suites, I can assure you that I can make a few picky rules for 
systems that would cause some of those systems to fail, but most end 
users who understood those few rules would say that those rules were 
silly. Of course, different end users have different requirements. It 
is generally felt by vendors that some/many of the rules for FIPS 140 
are silly and unneeded; on the other hand, it is likely that at least 
a few USGovt customers would really want some of those rules for 
particular reasons.

>Presumably a vendor submitting a product for evaluation believes 
>that the product is ready, and will pass, otherwise the vendor is 
>wasting money on the evaluation effort.

Wrong, very wrong. You cannot tell what the testing agency might try 
to knock you down on ahead of time. Of course, you cover what you 
can, but you also know that there is a good chance they'll just whack 
you a bit because it makes them look good.

>The fact that many products fail evaluation (for good security 
>reasons), tells me that a vendor's claim that a product "could be 
>evaluated" is not much of an assurance.

If you believe in the testing regime, that's fine. Others don't, and 
I believe for very good reasons in their own use scenarios.

--Paul Hoffman, Director
--VPN Consortium