[CFRG] Re: Where should test vectors live?

["D. J. Bernstein" <[e-mail]>]

Instead of defining formats for test vectors, I would suggest defining a
format for reference code (e.g., Python script meeting a specified API;
can have the format say which Python version). An obvious illustration
of using the format would be a general-purpose tool to create test
vectors given the reference code---and then there's no reason to post
the test vectors; people should run the tools locally. SUPERCOP already
generates tests for thousands of C implementations meeting the SUPERCOP
API, and there are examples online of how to do the same tests in
Python. SUPERCOP also supports generating implementation-specific tests;
this is sometimes useful for testing slow-to-find corner cases.

One advantage of this modularity is that implementors will easily be
able to swap in different testing tools (e.g., using fuzzers, or simply
testing more inputs) without being constrained by the limits of posted
test vectors. Of course, reference code is also useful for much more
than just testing.

---D. J. Bernstein