[CFRG] Re: Where should test vectors live?

["D. J. Bernstein" <[e-mail]>]

John Mattsson writes:
> Reference code only produces positive test vectors.

That depends on how the code is used! For example, after using KEM code
to generate legitimate ciphertexts, SUPERCOP automatically flips some
bits in the ciphertexts and tries decapsulating the results. There are
528 KEM implementations that have been submitted to and included in
SUPERCOP (e.g., 3 implementations of kyber1024), and these negative
tests are applied to all of them, without them having to do any work
beyond supporting the easy SUPERCOP API.

Meanwhile KEM teams following NIST's rules (and newer KEMs imitating
that pattern) have distributed endless copies of NIST's unmaintained
PQCgenKAT_kem.c (and the accompanying non-constant-time rng.c), along
with "KAT" files listing the resulting KEM test vectors. These KAT files
include only positive tests. How is that KAT ecosystem supposed to be
upgraded to add negative tests? Does someone patch PQCgenKAT_kem.c to
add those tests, and then try to convince everyone to remove the
existing KAT files in favor of the extended KAT files? Or should there
be separate NEGKAT files?

This is an illustration of how focusing on distribution of test vectors
ends up with lower-quality tests than focusing on distribution of (1)
reference code for each cryptographic primitive in a unified API and (2)
modular test tools that apply to anything using that API.

---D. J. Bernstein