[CFRG] Re: Where should test vectors live?

[Christopher Patton <[e-mail]>]

Dan,


Instead of defining formats for test vectors, I would suggest defining a
> format for reference code (e.g., Python script meeting a specified API;
> can have the format say which Python version). An obvious illustration
> of using the format would be a general-purpose tool to create test
> vectors given the reference code---and then there's no reason to post
> the test vectors; people should run the tools locally. SUPERCOP already
> generates tests for thousands of C implementations meeting the SUPERCOP
> API, and there are examples online of how to do the same tests in
> Python. SUPERCOP also supports generating implementation-specific tests;
> this is sometimes useful for testing slow-to-find corner cases.
>
> One advantage of this modularity is that implementors will easily be
> able to swap in different testing tools (e.g., using fuzzers, or simply
> testing more inputs) without being constrained by the limits of posted
> test vectors. Of course, reference code is also useful for much more
> than just testing.


The draft in question does have reference code (
https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/vdaf_poc) and
in fact we use this to generate test vectors and the spec itself (by
copy-pasting). The reference code is Python, but it requires SageMath to
run (e.g., `sage -python gen_test_vec.py` to generate test vectors). It
would be really cool to see implementers integrate the reference code
itself into their tests, though I think sage is too big a dependency to
carry around. We could probably remove it for this particular draft.

I think Python is by far the most common language for reference code at
CFRG. Some drafts require sage, others don't.

You may be interested in reviewing
https://github.com/cfrg/draft-irtf-cfrg-cryptography-specification.

Chris P.