Unit test proposal

Kenneth McMillan <kenmcmil@microsoft.com> Wed, 20 February 2019 01:05 UTC

Return-Path: <kenmcmil@microsoft.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4665F130E25 for <quic@ietfa.amsl.com>; Tue, 19 Feb 2019 17:05:37 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.001
X-Spam-Level:
X-Spam-Status: No, score=-2.001 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mu1Uxqb8wwhz for <quic@ietfa.amsl.com>; Tue, 19 Feb 2019 17:05:35 -0800 (PST)
Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-eopbgr790114.outbound.protection.outlook.com [40.107.79.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 23FEF128D0B for <quic@ietf.org>; Tue, 19 Feb 2019 17:05:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/LhgWvKuwjsPS4cPAo5/bfmbtkiniNcrTSdvW7xJ6MU=; b=bdsOZDoYZ8ct3+ORlAZ0+IMYne3LZKyDZ3X6v7KkjMwAx7uPwaAZEarv49O3FJ2yatW7Z2RGXk7X1hyuatO6Qn2PxHdjo6j/7/FKOAuU6De2DpdvPLCfLluFIyIEvydB9JMqh/b7qkwrvYuFB+zRQDhkrcf/yuZg3nr+h/pYqnc=
Received: from MW2PR2101CA0024.namprd21.prod.outlook.com (2603:10b6:302:1::37) by MWHPR21MB0750.namprd21.prod.outlook.com (2603:10b6:300:76::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.4; Wed, 20 Feb 2019 01:05:33 +0000
Received: from BL2NAM06FT005.Eop-nam06.prod.protection.outlook.com (2a01:111:f400:7e55::201) by MW2PR2101CA0024.outlook.office365.com (2603:10b6:302:1::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1665.0 via Frontend Transport; Wed, 20 Feb 2019 01:05:33 +0000
Received-SPF: Pass (protection.outlook.com: domain of microsoft.com designates 157.58.196.101 as permitted sender) receiver=protection.outlook.com; client-ip=157.58.196.101; helo=mail.exchange.microsoft.com;
Received: from mail.exchange.microsoft.com (157.58.196.101) by BL2NAM06FT005.mail.protection.outlook.com (10.152.106.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1679.0 via Frontend Transport; Wed, 20 Feb 2019 01:05:32 +0000
Received: from CYS-E16DFM04.exchange.corp.microsoft.com (10.221.226.71) by CYS-E16DFM02.exchange.corp.microsoft.com (10.221.226.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1713.2; Tue, 19 Feb 2019 17:05:31 -0800
Received: from CYS-E16DFM04.exchange.corp.microsoft.com ([fe80::79:73ff:fe2b:1f81]) by CYS-E16DFM04.exchange.corp.microsoft.com ([fe80::79:73ff:fe2b:1f81%17]) with mapi id 15.01.1713.002; Tue, 19 Feb 2019 17:05:31 -0800
From: Kenneth McMillan <kenmcmil@microsoft.com>
To: QUIC WG <quic@ietf.org>
Subject: Unit test proposal
Thread-Topic: Unit test proposal
Thread-Index: AQHUyLDljBly2P/KJkCJ6nDqOPEc3A==
Date: Wed, 20 Feb 2019 01:05:31 +0000
Message-ID: <e53f540dae3845659b99ea0c0626aeba@CYS-E16DFM04.exchange.corp.microsoft.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.221.255.3]
Content-Type: multipart/alternative; boundary="_000_e53f540dae3845659b99ea0c0626aebaCYSE16DFM04exchangecorp_"
MIME-Version: 1.0
X-EOPAttributedMessage: 0
X-MS-Office365-Filtering-HT: Tenant
X-Forefront-Antispam-Report: CIP:157.58.196.101; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(979002)(136003)(346002)(376002)(39860400002)(396003)(2980300002)(189003)(199004)(10090500001)(68736007)(478600001)(7116003)(71190400001)(561944003)(26005)(336012)(8936002)(54896002)(3846002)(356004)(186003)(6916009)(16586007)(6116002)(14454004)(55016002)(5660300002)(2906002)(97736004)(316002)(106466001)(53936002)(22452003)(476003)(81156014)(102836004)(8676002)(126002)(86362001)(81166006)(7696005)(86612001)(10290500003)(486006)(7736002)(19627405001)(108616005)(24736004)(106002)(6346003)(84326002)(3480700005)(426003)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR21MB0750; H:mail.exchange.microsoft.com; FPR:; SPF:Pass; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1;
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 89819397-30f9-4790-828f-08d696cf83a5
X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605104)(4608103)(4709054)(2017052603328)(7193020); SRVR:MWHPR21MB0750;
X-MS-TrafficTypeDiagnostic: MWHPR21MB0750:
X-O365-Sonar-Daas-Pilot: True
X-Microsoft-Exchange-Diagnostics: 1; MWHPR21MB0750; 20:dfrHUdlMjXdBEpbxb+/uBKoqDUJWhopbW3J7us4EH5SDiPmBIZIlNp5OQVD6CyNUOGaD8nEg5dwSi71jDvL1b41tWOtWiFfnU6aUSyoEMj8IOMy43VRHo5NDt29PbsbRVXRflp30zkShBlmyNiqyhHBweAkxIItA3UBNPvlWpAgpXIUzXiFlKw7cUAsSr3c7FDIREyq1Prjy0K69GqytMBsPWO9P7tha1L0Y0TeMOkESW70gLWxC+JrrPp/obLGe3GwOf9JMak5lsbdrZn+j41M5i3rwW9snBGOwmv93kAkUTUHVoSb+mA1/XNzwdl/jq5goh32fzfvqZx+0tzSlRV14xCsqdl+4xTwGuErPgjOGOljJzQylckNV352/6Yp+4iPXcIU2OFyZZYiXZmp7iYOsdpsrTE+q8bjywyCviy5P+XICBJ86vKqqonEhmuqspjy5Rs/m7f/oyzed+Z5AdM5/3qql/lDgZ2GM6ww5J9uJHdgbroz5pOMKRDEHLOS5
X-Microsoft-Antispam-PRVS: <MWHPR21MB07503D176E99025F0F9B069EC77D0@MWHPR21MB0750.namprd21.prod.outlook.com>
X-Forefront-PRVS: 0954EE4910
X-Microsoft-Exchange-Diagnostics: 1; MWHPR21MB0750; 23:TzkIxu57gok+RguI0cYp4fb8KYt0T103OrCDDmQlO4cAjggGH0k/qzvDHDg2EBl6ffpmy44F3M+KFKWnZyaAtZ/5B9vqUQbgCUMo/qa94z7Z5U39uRFkAjY7IR+TkKHAr/Rusf009pU1Oyccq0gpWknhblepbfWVAdvKGnnUrkRmm7q3BLfLNXIrj8ut0zRh0eG83SG4qR8t4tH5U9BAUcZ8JSLHYXZCJd6cxacd0OfWa2RDwG8KU4Fwss20oSnyVQIgrTYk/M71wezLZlsz03ULv0AB7OAq7axunk8H+w2rbu+pzMd06a3JM5eoyYt4R1ieznG1K16lHO+9gDxBVw7yn0xxi9/SS62WHSn3hSc7dS+48UvIGf0w3vABVZPpx0P33gD8UeNbCHcNS5t8y3LrIhsMI/oy8fr1BD+is01HaYbNz8/MT1qb8On1hAx9kIPts1/HcB6Lh2EZep7uSbbkLQXKD4zopP7f/gEEnySfWFOWtD2VfYrXVKKVcuSLkaE1hv9tAHVzKw0SZAoSfqnfAtxgFpqaMxevT07y3QZL8erN0F4upDQUZZdNSpGtDsei0AIHu/LMAVywKVSC5D5KjPtkoz+sZc/BblyJzQtjwWOnWkDfcXVEyY7QcCC4zAmfxHjpJN5Yma7n+gkZS4W908yKfLVqzoxi+SV8XgnCmXbJyxwWhl2QFcHJR7cWMilkMSuWpDr8jYv/JgJaxLVMVVjkYFUMMFfrOOphWK5M0SyoNh75RiTWaVvURvKWJW1oEWDpWTn2pEThSBmnggeadGw35g/GmKXiHlr2AfsYf/JxgSULeIPHeMg4wBc8gnkf8us8HQ3lkt6hloOxemtRhtFfl0UDgkiRJytT4b9g+wsldSRGysDj/J8gEntodgS0zNVkElZBlBfwoEsCiNjMSbcMqI5bHMTWGdWdgy8FA6ZqNscWzwX8J6Dc/7/z3gAjqfrVpHTwzX0MVQiL0XPD0daCTxfUdRwkZZmVDyuxj8BWAy5aF7awOolIeK2fzQSJMlAsq2kWl3Gdw3rdCduG9Gxmvq57wDPZQcj8tc2tKNm+G3H5p9An1A9nMwRQZQyEZRdxu9tWrR/dNAFnAbyfpHLMGAylq+GuavC5kP9FvqAlaFoSJuOABJxYnivQUREO+o18VTtwQRWh9qyng8DSE2GI+dXRISo9iVmBlO8B51n+15Fit9Ye1krf1MhFJIwm+PqELaAsSG7Z2d5mcmau7vs80XqhPqHuJzNIE3llVqlTih3lJ5n4L2wyGCncemAyboAXFtxhYvxQzLvWvA==
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam-Message-Info: 44zIA597WOimU+QT0jwALvuzcCym0WuzoUF9qCyjaW3K4+hhlDv1BgL2ytcM5tfUCru9WEoKXJ94UrNfgI16MQFl8N6nv4gD+dFim+jIYwTKbS49Wp8amCa1Im8Cf1tl5YxLpoSgJCKL6P7BvIZhROSR9bKO7i8K+VuQu3Wra7mxv3vznENZrUI7ey2vEqY33gy5j5cBXnzoJ2BVtNaD8x5E1XV6ONYqguSgjSUdIOA46I6Ci4dRXF0TsOT0Hxd6Hgt8TjikWQ2Rneo12LzQ8rk8s9pO70aC/+++D0wfR94Yt6ec3TSOcT8vTvc5Xp9lYdcxmmA0pNZbBX4QVTtGDCjiup5HsIYVaLMuf4yZMXmbfGSh22KXye0+Bkw8DHsooBkcNaHn9Hm14/LylLwWi3XXeLLuN8YdJa1lfuTdWkk=
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2019 01:05:32.9534 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 89819397-30f9-4790-828f-08d696cf83a5
X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47; Ip=[157.58.196.101]; Helo=[mail.exchange.microsoft.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0750
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/XNfmMQzhr87vn0kJ0Se6mZ9uQjs>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Feb 2019 01:05:37 -0000

I have a proposal to make regarding unit testing of QUIC implementations. I would like to ask if there is any interest in the group and if anything along these lines has been considered or is in the works.

Currently I'm working on unit testing of QUIC implementations, but the effort is somewhat hampered by the fact that they all have different API's. That is, they present different interfaces to applications, and also use different system service API's. As a result, I can only test the implementations through the network (since this interface is standardized). This means that there are many behaviors that cannot be tested, since they can only stimulated through the implementation's client or system API's.

What I would like to propose is a common unit test interface for QUIC. This would be a value-based, language-independent API that provides QUIC services to applications and system services to QUIC. This interface would have several advantages:

1) Since it is generic, it would allow re-use of unit tests between implementations
2) Since it is value-based, it would allow logging and replay of runs
3) It could be made to support deterministic replay even for threaded implementations, by simulating cooperative multitasking.
4) It could also be used in interoperability testing to provide local replay.

In my experience, logging and replay of an interface is also enormously useful for bug reports, since there is no need even to build the user's application to diagnose the problem.

The investment required would be that each implementation would need a wrapper that presents the common unit test interface. My sense is that this effort would be moderate, since QUIC's interface to applications is relatively simple, and the implementations seem to require few system services (mainly file I/O, network I/O, locks, timers and random number generation). I think the advantages mentioned above would more than pay back this effort.

I'm not thinking of this as an official standard. It would just be a de facto standard that would be used optionally by developers who want to take advantage of a common base of unit tests.

Please let me know if you think this seems worthwhile, or if it's redundant with any existing work.


Thanks -- Ken