From: John Mattsson <john.mattsson@ericsson.com>
To: Christopher Patton <cpatton=40cloudflare.com@dmarc.ietf.org>,
	"cfrg@irtf.org" <cfrg@irtf.org>
Thread-Topic: [CFRG] Re: Where should test vectors live?
Thread-Index: 
 AQHbCvkbomwU7QduekSKv3Exw7kAirJgfpyAgABxMYCAAARuAIAAAvkAgAAi2QCAAC0VgIAAZsS/
Date: Sat, 21 Sep 2024 04:58:07 +0000
Message-ID: 
 <GVXPR07MB96788E2243EC05BF3A8E6DBF896D2@GVXPR07MB9678.eurprd07.prod.outlook.com>
References: 
 <CAG2Zi22ya_6h1Q_rs9Tf6tJ3qao-X2_wWhCgxoXD3OQnf+MOUw@mail.gmail.com>
 <20240920200220.402868.qmail@cr.yp.to>
 <CAG2Zi21QNTZ8rZ-j5nrkbRbrk4pdknXjdsCiLip30HjvweK9HA@mail.gmail.com>
In-Reply-To: 
 <CAG2Zi21QNTZ8rZ-j5nrkbRbrk4pdknXjdsCiLip30HjvweK9HA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-GB
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: 
 =?us-ascii?Q?EL6TUsjzdSknk2dbefdkNsBEiqt+y9Aj9iKHVprqNlOfwymbFaRcr/8Aw2cz?=
 =?us-ascii?Q?N26yOi3VaYt5PRtVNfKc2iowHS+2/foWUzHHfYCifBzQiQI3mWFqtAbiKoOk?=
 =?us-ascii?Q?W+pWIz57LkZC47n3crzFKXXubuItfKpZVI0d4Jv9i+WR1trg3z0ome4AbMA4?=
 =?us-ascii?Q?TqthzqGM9IX1PiuI0Y+TgDbT7mSyP2LHwu4OC4YfLhTmoDnJ9CoZLV+K8PVN?=
 =?us-ascii?Q?yq0f16ahC6LdmfP81RtmCr3FJnWin0PEV2sVVYw3dwuhFnTbj6ZSkY/93OUb?=
 =?us-ascii?Q?hfa17HXlT47/b/20yjMEKGMPzynUZHGYd8lgCoj5sSMb/+M8uTyGBOh3w/t3?=
 =?us-ascii?Q?Ur+i4wcgUvKWoSjcugrbePP2kvaErWRAcTHbLAlgX9Ks0Dj2fb+yJP+Hproz?=
 =?us-ascii?Q?LY4ux3dAH4x83Cu/nLboU0xANXYWzMeq+AeXU1PDHPoDZLZDfrEja7fxJbtT?=
 =?us-ascii?Q?RwXK79vN/hTTO+fyOSpo4rRdJwT7xdsZxkPuQy4dR9IFldlHIPqCGnmeAVYq?=
 =?us-ascii?Q?aALw5ChVdJW5Z/6jfQ6HzJucrEAaYcaY0lM71SSxskP27zwfwepvV0oEYh7Z?=
 =?us-ascii?Q?kdD9QmzxKyDP/dt2vaAUEwdxUktI29gQz3J5cN7Hwris66oP+H4YHCzqDQjg?=
 =?us-ascii?Q?tkzGPz6Bw2elsnjn7+4O4VdY2rUWvDWLTz68ty60sNmpiBpQL0CppVyGrZk9?=
 =?us-ascii?Q?1reaOIqSz5SdLD+ocm1kFUZIVigqOGyrU5/RFwspm7u/GESfExgYjqpubtE4?=
 =?us-ascii?Q?mrUFtfktE7lXDieO5+USw60OrnzKttMkbB/+MRC0kQwc/Bqoym16WOSVgD6a?=
 =?us-ascii?Q?hsx7X/UwIV3feu0B5tMvNFyEjsl2xhUM6a+pKhVXwFT2B6aWKfDQcKzkPTDq?=
 =?us-ascii?Q?iop4rSd4mxrqScLxtzqFPYSE1/CTrOzdExCcMRXRWmfVDx+wD4mYIHUYMVvc?=
 =?us-ascii?Q?JmwZOe5zADrn47Mip8f6OrK/yaO91tddPy/iVn//iP1M8GVXeGCbe2t7t5zM?=
 =?us-ascii?Q?5+AZPgnGNtrBGERw00zpHb5WjOf/cUAnanA8m22f4Mza5+jLKsbPKawPgIVd?=
 =?us-ascii?Q?aES3oWHn7novKm1i8huVWpGY0t9ZoULOdkatCw7z/Sfi0h1fjkP0OFii1P2u?=
 =?us-ascii?Q?w6kGEz7ExVcuZuZtDI2nChYDVQg5GBeihIIn8riuoc3CSjOyFtzCl4bgwDSD?=
 =?us-ascii?Q?iK3gdKta+Lkx8RteYHuoOgoyH0Oniiml9uGDJR5Ha3B5l/f6ZmwhFY4K5Vb6?=
 =?us-ascii?Q?C9NGmTXiQCUbeTiw+Yb9bEnTrP2O59bk0ml/jlJbj9ya7g4KrUdXMptIeJp2?=
 =?us-ascii?Q?Ae5T+X9WGbZuBTYMsgaEu6wVmUyxuSRlyPMdUDu49rV0TesmucAJvvM+3N/0?=
 =?us-ascii?Q?fLEf+DuyGZwXxf+yYhyCwD6wH7mGhQrPgYd7nR/juAbiqalJa1Hb/Bi/fC9T?=
 =?us-ascii?Q?2gxGZUmq9VscD9myFjLQpW6gKqcyGhDCyh3KEoXpznNcV2T3QaNzvHJoLCPW?=
 =?us-ascii?Q?czTB09AZ9SpwDyJl3Ah3hY0icW+uLTzoAC5tyDzxF9JqSm51ODMdleDyDh5g?=
 =?us-ascii?Q?73ySgvaEDz9277j83dflXiJuOUNpY06m5IPlX0hYZ4dcFukqgBK4AD1j8MTi?=
 =?us-ascii?Q?ToX484LYuPGFRpCyDgw3seM=3D?=
Content-Type: multipart/alternative;
	boundary="_000_GVXPR07MB96788E2243EC05BF3A8E6DBF896D2GVXPR07MB9678eurp_"
MIME-Version: 1.0
X-OriginatorOrg: ericsson.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: GVXPR07MB9678.eurprd07.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 f4239d35-b86a-453b-ee8e-08dcd9f9fb9a
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Sep 2024 04:58:07.0809
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 
 ypukte71EON8PRrOE9sLreMVeRtI/ETsLd+IPIcGJXVHE3ejqL+oIdnXqo2+GO4eBNLE/uhmFKMINj12sbvCTZuX+pB1P+xmrWGVnNxwSu4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR07MB9917
Message-ID-Hash: FQZK3WY64WADFTZOW5VQIXRNG2HRMDKV
X-Message-ID-Hash: FQZK3WY64WADFTZOW5VQIXRNG2HRMDKV
X-MailFrom: john.mattsson@ericsson.com
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency;
 loop; banned-address; member-moderation; header-match-cfrg.irtf.org-0;
 nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size;
 news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.9rc4
Precedence: list
Subject: =?utf-8?q?=5BCFRG=5D_Re=3A_Where_should_test_vectors_live=3F?=
List-Id: Crypto Forum Research Group <cfrg.irtf.org>
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/cfrg/VJQMAvElCUk050C7rkXdG-X7oTk>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cfrg>
List-Help: <mailto:cfrg-request@irtf.org?subject=help>
List-Owner: <mailto:cfrg-owner@irtf.org>
List-Post: <mailto:cfrg@irtf.org>
List-Subscribe: <mailto:cfrg-join@irtf.org>
List-Unsubscribe: <mailto:cfrg-leave@irtf.org>

--_000_GVXPR07MB96788E2243EC05BF3A8E6DBF896D2GVXPR07MB9678eurp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

D. J. Bernstein wrote:
>Instead of defining formats for test vectors, I would suggest defining a
>format for reference code (e.g., Python script meeting a specified API;
>can have the format say which Python version). An obvious illustration
>of using the format would be a general-purpose tool to create test
>vectors given the reference code---and then there's no reason to post
>the test vectors; people should run the tools locally.

Reference code only produces positive test vectors. Unless all inputs are v=
alid, negative test vectors are essential for catching non-compliant implem=
entations such as ECC implementations not doing point validation. Negative =
test vectors typically have to be hand crafted.

Cheers,
John

From: Christopher Patton <cpatton=3D40cloudflare.com@dmarc.ietf.org>
Date: Saturday, 21 September 2024 at 00:44
To: cfrg@irtf.org <cfrg@irtf.org>
Subject: [CFRG] Re: Where should test vectors live?

Dan,


Instead of defining formats for test vectors, I would suggest defining a
format for reference code (e.g., Python script meeting a specified API;
can have the format say which Python version). An obvious illustration
of using the format would be a general-purpose tool to create test
vectors given the reference code---and then there's no reason to post
the test vectors; people should run the tools locally. SUPERCOP already
generates tests for thousands of C implementations meeting the SUPERCOP
API, and there are examples online of how to do the same tests in
Python. SUPERCOP also supports generating implementation-specific tests;
this is sometimes useful for testing slow-to-find corner cases.

One advantage of this modularity is that implementors will easily be
able to swap in different testing tools (e.g., using fuzzers, or simply
testing more inputs) without being constrained by the limits of posted
test vectors. Of course, reference code is also useful for much more
than just testing.

The draft in question does have reference code (https://github.com/cfrg/dra=
ft-irtf-cfrg-vdaf/tree/main/poc/vdaf_poc), and in fact we use this to gener=
ate test vectors and the spec itself (by copy-pasting). The reference code =
is Python, but it requires SageMath to run (e.g., `sage -python gen_test_ve=
c.py` to generate test vectors). It would be really cool to see implementer=
s integrate the reference code itself into their tests, though I think sage=
 is too big a dependency to carry around. We could probably remove it for t=
his particular draft.

I think Python is by far the most common language for reference code at CFR=
G. Some drafts require sage, others don't.

You may be interested in reviewing https://github.com/cfrg/draft-irtf-cfrg-=
cryptography-specification.

Chris P.


--_000_GVXPR07MB96788E2243EC05BF3A8E6DBF896D2GVXPR07MB9678eurp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Aptos;
	panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	font-size:12.0pt;
	font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Aptos",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;
	mso-ligatures:none;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
	{page:WordSection1;}
--></style>
</head>
<body lang=3D"en-SE" link=3D"blue" vlink=3D"purple" style=3D"word-wrap:brea=
k-word">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">D. J. Bernstein wrote:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">&gt;Instead of defining formats for test vectors, I =
would suggest defining a<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">&gt;format for reference code (e.g., Python script m=
eeting a specified API;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">&gt;can have the format say which Python version). A=
n obvious illustration<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">&gt;of using the format would be a general-purpose t=
ool to create test<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">&gt;vectors given the reference code---and then ther=
e's no reason to post<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">&gt;the test vectors; people should run the tools lo=
cally.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">Reference code only produces positive test vectors. =
Unless all inputs are valid, negative test vectors are essential for catchi=
ng non-compliant implementations such
 as ECC implementations not doing point validation. Negative test vectors t=
ypically have to be hand crafted.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-size:11.0pt;mso-f=
areast-language:EN-US">John<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"><o:p>&nbsp;</o:p></span></p>
<div id=3D"mail-editor-reference-message-container">
<div>
<div>
<div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm =
0cm 0cm">
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><b><span style=3D"col=
or:black">From:
</span></b><span style=3D"color:black">Christopher Patton &lt;cpatton=3D40c=
loudflare.com@dmarc.ietf.org&gt;<br>
<b>Date: </b>Saturday, 21 September 2024 at 00:44<br>
<b>To: </b>cfrg@irtf.org &lt;cfrg@irtf.org&gt;<br>
<b>Subject: </b>[CFRG] Re: Where should test vectors live?<o:p></o:p></span=
></p>
</div>
<div>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Dan,<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<div>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0c=
m 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class=3D"MsoNormal">Instead of defining formats for test vectors, I woul=
d suggest defining a<br>
format for reference code (e.g., Python script meeting a specified API;<br>
can have the format say which Python version). An obvious illustration<br>
of using the format would be a general-purpose tool to create test<br>
vectors given the reference code---and then there's no reason to post<br>
the test vectors; people should run the tools locally. SUPERCOP already<br>
generates tests for thousands of C implementations meeting the SUPERCOP<br>
API, and there are examples online of how to do the same tests in<br>
Python. SUPERCOP also supports generating implementation-specific tests;<br=
>
this is sometimes useful for testing slow-to-find corner cases.<br>
<br>
One advantage of this modularity is that implementors will easily be<br>
able to swap in different testing tools (e.g., using fuzzers, or simply<br>
testing more inputs) without being constrained by the limits of posted<br>
test vectors. Of course, reference code is also useful for much more<br>
than just testing.<o:p></o:p></p>
</blockquote>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class=3D"MsoNormal">The draft in question does have reference code (<a h=
ref=3D"https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/vdaf_poc"=
>https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/vdaf_poc</a>), =
and in fact we use this to generate
 test vectors and the spec itself (by copy-pasting). The reference code is =
Python, but it requires SageMath to run (e.g., `sage -python gen_test_vec.p=
y` to generate test vectors). It would be really cool to see implementers i=
ntegrate the reference code itself
 into their tests, though I think sage is too big a dependency to carry aro=
und. We could probably remove it for this particular draft.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class=3D"MsoNormal">I think Python is by far the most common language fo=
r reference code at CFRG. Some drafts require sage, others don't.<o:p></o:p=
></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class=3D"MsoNormal">You may be interested in reviewing <a href=3D"https:=
//github.com/cfrg/draft-irtf-cfrg-cryptography-specification">
https://github.com/cfrg/draft-irtf-cfrg-cryptography-specification</a>.<o:p=
></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Chris P.<o:p></o:p></p>
</div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

--_000_GVXPR07MB96788E2243EC05BF3A8E6DBF896D2GVXPR07MB9678eurp_--