[CFRG] Re: Where should test vectors live?

[John Mattsson <[e-mail]>]

D. J. Bernstein wrote:
>Instead of defining formats for test vectors, I would suggest defining a
>format for reference code (e.g., Python script meeting a specified API;
>can have the format say which Python version). An obvious illustration
>of using the format would be a general-purpose tool to create test
>vectors given the reference code---and then there's no reason to post
>the test vectors; people should run the tools locally.

Reference code only produces positive test vectors. Unless all inputs are valid, negative test vectors are essential for catching non-compliant implementations such as ECC implementations not doing point validation. Negative test vectors typically have to be hand crafted.

Cheers,
John

From: Christopher Patton <cpatton=[e-mail]>
Date: Saturday, 21 September 2024 at 00:44
To: [e-mail] <[e-mail]>
Subject: [CFRG] Re: Where should test vectors live?

Dan,


Instead of defining formats for test vectors, I would suggest defining a
format for reference code (e.g., Python script meeting a specified API;
can have the format say which Python version). An obvious illustration
of using the format would be a general-purpose tool to create test
vectors given the reference code---and then there's no reason to post
the test vectors; people should run the tools locally. SUPERCOP already
generates tests for thousands of C implementations meeting the SUPERCOP
API, and there are examples online of how to do the same tests in
Python. SUPERCOP also supports generating implementation-specific tests;
this is sometimes useful for testing slow-to-find corner cases.

One advantage of this modularity is that implementors will easily be
able to swap in different testing tools (e.g., using fuzzers, or simply
testing more inputs) without being constrained by the limits of posted
test vectors. Of course, reference code is also useful for much more
than just testing.

The draft in question does have reference code (https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/vdaf_poc) and in fact we use this to generate test vectors and the spec itself (by copy-pasting). The reference code is Python, but it requires SageMath to run (e.g., `sage -python gen_test_vec.py` to generate test vectors). It would be really cool to see implementers integrate the reference code itself into their tests, though I think sage is too big a dependency to carry around. We could probably remove it for this particular draft.

I think Python is by far the most common language for reference code at CFRG. Some drafts require sage, others don't.

You may be interested in reviewing https://github.com/cfrg/draft-irtf-cfrg-cryptography-specification.

Chris P.