Re: [saag] Algorithms/modes requested by users/customers

[[e-mail] (Peter Gutmann)]

Randall Atkinson <[e-mail]> writes:
>Earlier, Peter Gutmann wrote:
>% If FIPS 140 is the answer now, why wasn't the Orange Book
>% the answer then?
>
>You are comparing apples to oranges above.
>
>FIPS-140 is only about assurance for cryptographic modules.
>Orange Book (TCSEC) was only about operating system security.
>
>The two address different issues.

They certainly seem to be addressing them in very similar ways.

>Or were you proposing to setup a monopoly ?

Ten years ago the rule-of-thumb estimate for a FIPS 140 level 1 eval was $100K
and 6-12 months no matter who you went to.  In 2006 (last time I checked) the
rule-of-thumb estimate for a FIPS 140 level 1 eval was $100K and 6-12 months
no matter who you went to.  So it looks like we already have a well-
established oligopoly with FIPS 140 evals, the only difference is the
nomenclature.

>One needs a process that is as consistent and reproducible as practical

Why?  This is the ISO 9000-for-software fallacy, reproducibility is great when
you're cranking out lightbulbs, but not much use for software, which isn't a
mass production process but instead is based on the cloning of the result of a
one-off development effort which is the product of the creativity, skill, and
co-operation of developers and users.  I can easily get 100% consistency in
reproducing an executable on a CDROM, but I don't think that's what's meant
here.

>If you think that FIPS-140-* is a target-rich environment, then please try to
>seriously propose something better.  I understand NIST and its partners are
>looking to evolve into FIPS 140-3 from FIPS 140-2.

It depends on what you want from FIPS 140 (and I'm talking specifically FIPS
140 for software modules here, which is the form that 99.9% of users encounter
it in ).  Some people have said you get "assurance", but the assurance you're
getting here is a high level of assurance that the vendors were desperate
enough for government sales that they shelled out a large amount of money in
order to get a ticket to ride the gravy train.

The problem is that like "religion" or "morals", "assurance" is a loaded term
and can be interpreted to mean almost anything.  Without some basic definition
of what's meant by "assurance", it's not really possible to reason about this.
So here's an attempt to nail it down: For me (and I would guess most end
users) assurance is being able to sleep at night knowing there's little chance
of anyone compromising my system ("things that should happen do happen, things
that shouldn't happen don't happen").

Let's see what FIPS 140 gives you to help you sleep at night.  It tests some
crypto mechanisms, but ignores others.  If you're buying an IPsec product it
doesn't test all the mechanisms needed for IPsec.  If you're buying a TLS
product it doesn't test the mechanisms needed for TLS.  If you're buying...
well, you get the idea.

Compare this to the example I gave earlier of performing a TLS exchange with
Amazon.  This performs an in-depth test of all the crypto algorithms
(corresponding to the FIPS algorithm tests, including ones that FIPS ignores),
and the crypto mechanisms (many/most of which FIPS again ignores).  In other
words simply by connecting to Amazon using TLS and ordering a "Scrubs" DVD for
$19.95 I'm getting more comprehensive algorithm testing than I can for a five-
figure sum with the FIPS algorithm tests.

So what else do you get?  You get a bit of a code review and some checking
that you don't do stupid things with keys.  Only some of the code is reviewed
(for example that trivial buffer overflow in the client handshake that allows
a remote attacker to do whatever they want with your server, including the
FIPS 140-evaluated portion, isn't checked because it's outside the scope of
what FIPS 140 is interested in), and (from having gone through several of
these) the review itself seems to be more coding style nitpicks than looking
for potential exploits, and in any case you can often document away any
problems without having to fix them.

So the assurance a FIPS 140 eval gets you is a tautological "the product
vendor was able to pass a FIPS 140 eval".  In terms of being able to
demonstrate this type of assurance, I agree completely with you that there is
no better mechanism for it than FIPS 140.

Admittedly there is some value to FIPS 140: You know that a subset of the
crypto used is OK, that the code doesn't do anything stupid with its keys
(unless the vendor has documented away the stupid things that do happen, e.g.
CryptoAPI's plaintext private-key export, you ask it "Give me the user's
private key in plaintext form" and it says "Sure, here you go"), and that it's
had a bit of a code check (and obviously if you're doing hardware crypto you
get a lot more value than this, but the discussion here is about software
modules).

What you don't get from FIPS 140 is the stuff that customers really care
about: Assurance that your IPsec product can actually do IPsec, that your TLS
product can do TLS, or that the product really is secure(-ish).  Excusing this
by saying that a FIPS 140 crypto product eval doesn't have to check half your
crypto so this isn't a problem is like saying that your TSA passenger-
screening process is OK even though it doesn't bother checking men with beards
or anyone under 5'6" tall.

If I'm going to spend $100K to feel safe then I can really think of better
ways than FIPS 140 (or CC, or ITSEC, or ...).  How about the following
comparison: You get to spend the cost of a FIPS 140 eval in one of two ways:

$100K: FIPS 140

-- or --

$20K: Fortify (or whatever) scan.
$20K: Cigital (or whatever) audit/analysis (these are just representative
      figures/estimates since there don't seem to be any fixed rates for this
      sort of thing).
$5K: Grant to French university to pen-test code "written at a German
     university".
$5K: Grant to British university to pen-test code "written at a French
     university".
$5K: Set up a fake banking site running the code and bring it to the attention
     of the Russian Business Network.
$free: Leak "security code for Windows 7" to Slashdot.
$free: Leak "security code for BluRay++" on warez boards.
$free: Leak "Diebold voting machine security code" to Cryptome.
$45K: 50" plasma TV, home theatre system, and beer to kill time while you
      wait for the results to come in (and to emphasise how much cheaper
      this is than FIPS 140 :-).

If your life depended on the security of the product that went through those
two processes, which one would you trust?  FIPS 140 is just not cost-effective
in providing assurance - it costs too much, it takes too long, and it doesn't
evaluate the stuff that real users care about having evaluated.

>The claims (not by me so much as by other folks on the SAAG list) have been
>that (1) FIPS-140 is better than other extant security evaluations

Thus showing that the politician's fallacy is alive and well :-).

There was a long and extremely interesting discussion on the (somewhat
misnamed) open-source list recently, thread topic "How can high assurance
FLOSS development be encouraged?".  The archives are at
http://lists.csl.sri.com/mailman/listinfo/open-source, but you have to sign up
to the list in order to access them.

Peter.