MIME-Version: 1.0
References: 
 <CAG2Zi22ya_6h1Q_rs9Tf6tJ3qao-X2_wWhCgxoXD3OQnf+MOUw@mail.gmail.com>
 <20240920200220.402868.qmail@cr.yp.to>
In-Reply-To: <20240920200220.402868.qmail@cr.yp.to>
From: Christopher Patton <cpatton@cloudflare.com>
Date: Fri, 20 Sep 2024 15:43:41 -0700
Message-ID: 
 <CAG2Zi21QNTZ8rZ-j5nrkbRbrk4pdknXjdsCiLip30HjvweK9HA@mail.gmail.com>
To: cfrg@irtf.org
Content-Type: multipart/alternative; boundary="000000000000f25baf062294c56b"
Message-ID-Hash: UV3R4MD47IR2LL2DQNGPPOW44CRREV4A
Precedence: list
Subject: =?utf-8?q?=5BCFRG=5D_Re=3A_Where_should_test_vectors_live=3F?=
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/cfrg/ezlx-v3e4p7pgCglEDXqTeTiYjg>

--000000000000f25baf062294c56b
Content-Type: text/plain; charset="UTF-8"

Dan,


Instead of defining formats for test vectors, I would suggest defining a
> format for reference code (e.g., Python script meeting a specified API;
> can have the format say which Python version). An obvious illustration
> of using the format would be a general-purpose tool to create test
> vectors given the reference code---and then there's no reason to post
> the test vectors; people should run the tools locally. SUPERCOP already
> generates tests for thousands of C implementations meeting the SUPERCOP
> API, and there are examples online of how to do the same tests in
> Python. SUPERCOP also supports generating implementation-specific tests;
> this is sometimes useful for testing slow-to-find corner cases.
>
> One advantage of this modularity is that implementors will easily be
> able to swap in different testing tools (e.g., using fuzzers, or simply
> testing more inputs) without being constrained by the limits of posted
> test vectors. Of course, reference code is also useful for much more
> than just testing.


The draft in question does have reference code (
https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/vdaf_poc), and
in fact we use this to generate test vectors and the spec itself (by
copy-pasting). The reference code is Python, but it requires SageMath to
run (e.g., `sage -python gen_test_vec.py` to generate test vectors). It
would be really cool to see implementers integrate the reference code
itself into their tests, though I think sage is too big a dependency to
carry around. We could probably remove it for this particular draft.

I think Python is by far the most common language for reference code at
CFRG. Some drafts require sage, others don't.

You may be interested in reviewing
https://github.com/cfrg/draft-irtf-cfrg-cryptography-specification.

Chris P.

--000000000000f25baf062294c56b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><div>Dan,</div><div><br></div><=
div><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr"><br></=
div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">Instead of defining f=
ormats for test vectors, I would suggest defining a<br>
format for reference code (e.g., Python script meeting a specified API;<br>
can have the format say which Python version). An obvious illustration<br>
of using the format would be a general-purpose tool to create test<br>
vectors given the reference code---and then there&#39;s no reason to post<b=
r>
the test vectors; people should run the tools locally. SUPERCOP already<br>
generates tests for thousands of C implementations meeting the SUPERCOP<br>
API, and there are examples online of how to do the same tests in<br>
Python. SUPERCOP also supports generating implementation-specific tests;<br=
>
this is sometimes useful for testing slow-to-find corner cases.<br>
<br>
One advantage of this modularity is that implementors will easily be<br>
able to swap in different testing tools (e.g., using fuzzers, or simply<br>
testing more inputs) without being constrained by the limits of posted<br>
test vectors. Of course, reference code is also useful for much more<br>
than just testing.</blockquote><div><br></div><div>The draft in question do=
es have reference code (<a href=3D"https://github.com/cfrg/draft-irtf-cfrg-=
vdaf/tree/main/poc/vdaf_poc">https://github.com/cfrg/draft-irtf-cfrg-vdaf/t=
ree/main/poc/vdaf_poc</a>), and in fact we use this to generate test vector=
s and the spec itself (by copy-pasting). The reference code is Python, but =
it requires SageMath to run (e.g., `sage -python gen_test_vec.py` to genera=
te test vectors). It would be really cool to see implementers integrate the=
 reference code itself into their tests, though I think sage is too big a d=
ependency to carry around. We could probably remove it for this particular =
draft.</div><div><br></div><div>I think Python is by far the most common la=
nguage for reference code at CFRG. Some drafts require sage, others don&#39=
;t.</div><div><br></div><div>You may be interested in reviewing <a href=3D"=
https://github.com/cfrg/draft-irtf-cfrg-cryptography-specification">https:/=
/github.com/cfrg/draft-irtf-cfrg-cryptography-specification</a>.</div><div>=
<br></div><div>Chris P.<br></div><br></div></div></div>

--000000000000f25baf062294c56b--