Re: An IETF repository for working code in our protocols?

Mark Nottingham <mnot@mnot.net> Thu, 20 August 2020 01:42 UTC

Return-Path: <mnot@mnot.net>
X-Original-To: wgchairs@ietfa.amsl.com
Delivered-To: wgchairs@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E02AC3A10E6 for <wgchairs@ietfa.amsl.com>; Wed, 19 Aug 2020 18:42:48 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level:
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=mnot.net header.b=dzYZJ/KH; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=CyRwUSDw
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vIO6CKElmPYQ for <wgchairs@ietfa.amsl.com>; Wed, 19 Aug 2020 18:42:46 -0700 (PDT)
Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 078743A1086 for <wgchairs@ietf.org>; Wed, 19 Aug 2020 18:42:41 -0700 (PDT)
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 4B2C35C0092; Wed, 19 Aug 2020 21:42:41 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Wed, 19 Aug 2020 21:42:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mnot.net; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm3; bh=i 7gnfkTQxzKbHqE9iwLylvlOEEa2FM8wvGrmIoKKyao=; b=dzYZJ/KHxHYDZURHi 6w+VnbvchZ5cdLI0iptGP43GIFq1LQGFBDkVfp3k9YksO5yzCEhsIuQhFyTwqLv7 jZjCv1DCpTBAuI5UPeGMmsCywgpdtnt90EE3XFh7eMpZjLPOipCFk4o6gbGyDWaB VSOehYiF+jq8uwFH0Ot5AojO7Tk9XUP0KOQMERj82goarABXxeqRY7c9H8/hcnS5 KhP+Xx5qQT0aHJu/cEscmbbHR9oq5f40nnEysSRNe4lXvI0d72dzgwX18eh6Xzgn v2HvO7ksF4qEHE12GzFrg5MjIiObH2/w/9lO7gjKSqx10MlKnhskGUtFCOi8hB2/ qaKxg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=i7gnfkTQxzKbHqE9iwLylvlOEEa2FM8wvGrmIoKKy ao=; b=CyRwUSDwuDxKQkApoiWUBzxDV1oBdKWL0o8SY5mDxdRUhc2mySOmCipe8 S6UbZ5MBLgE4nCUK9jDc4owzsVtfNAaLtAFHBmj8jys06yiyzJ3Ryrd54QalVfsz Cqr2jHokcqrEP4V1Di/pXdd1AQno+BdUHbbfQrkwldc7O/OqWNESJ+JsH4lhOVK9 W3chnZWHXFQTztIfRCh7N2Y8/TP3h70QpnGhvUC4auTD0VFRuqKcoVUGdCs11BKe D/kcDHBg4zNo6qbw3TAswbGjVfKjbw1m4APZU+qJJFMBx27O6ZbV/tyfh9EqU536 TospdFRLiQMTBvZ820HC6+EwngFMA==
X-ME-Sender: <xms:ENU9X8F_O937PbSXSRdR32Ck5isuOYEa8PjYgyQllKZTtSd7_hCMug>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedruddtledgfeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne goufhushhpvggtthffohhmrghinhculdegledmnecujfgurheptggguffhjgffgffkfhfv ofesthhqmhdthhdtvdenucfhrhhomhepofgrrhhkucfpohhtthhinhhghhgrmhcuoehmnh hothesmhhnohhtrdhnvghtqeenucggtffrrghtthgvrhhnpeejkeeigfegkeevudfhfeel teefkeehhfdtjeevtdehhedugfdvveevleehjeejhfenucffohhmrghinhepghhoohhglh gvrdgtohhmpdhgihhthhhusgdrtghomhdpihgrsgdrohhrghdpihgvthhfrdhorhhgpdhi vggvvgdrohhrghdprggtmhdrohhrghdpmhhnohhtrdhnvghtnecukfhppeduudelrdduje drudehkedrvdehudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpehmnhhothesmhhnohhtrdhnvght
X-ME-Proxy: <xmx:ENU9X1Xn_hIHWJ6yV8cyTOT9YK0ZBCXTqfQkUjHMmJKDy-MJ-sMZCw> <xmx:ENU9X2I1GwpOlRhlrhOfqqxgfZSuoeGSQlaT0nsvL9s6KY7G4BEn7g> <xmx:ENU9X-Eyxza1aAaARmmyeqX6QWkOnS-Bu4FsWaISHPPQ5OHNKoKz1A> <xmx:EdU9X1dZI_3tJV8qGoI9aMumIj7j7R9SOrt_rj36ewJ8OujlIsbJrw>
Received: from [192.168.7.30] (119-17-158-251.77119e.mel.static.aussiebb.net [119.17.158.251]) by mail.messagingengine.com (Postfix) with ESMTPA id 3B5F830600B1; Wed, 19 Aug 2020 21:42:38 -0400 (EDT)
Content-Type: text/plain; charset="us-ascii"
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: An IETF repository for working code in our protocols?
From: Mark Nottingham <mnot@mnot.net>
In-Reply-To: <3eb249f3-121f-4ef2-8963-d5e4534bb6bc@dogfood.fastmail.com>
Date: Thu, 20 Aug 2020 11:42:36 +1000
Cc: wgchairs@ietf.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <5096712F-013C-4F29-8FE7-7EFAC51037EB@mnot.net>
References: <CAMMTW_+Di=ZBJFLNPaVK6f3w3Yq-V-qau8G_rfGt96SX_aYAAA@mail.gmail.com> <8193D927-DDA8-4C74-BBD3-1AF6C9AFE98B@mnot.net> <3eb249f3-121f-4ef2-8963-d5e4534bb6bc@dogfood.fastmail.com>
To: Gondwana Bron <brong@fastmailteam.com>
X-Mailer: Apple Mail (2.3608.80.23.2.2)
Archived-At: <https://mailarchive.ietf.org/arch/msg/wgchairs/rfQLF3iuXJsHy-zoOyEKn_r5G28>
X-BeenThere: wgchairs@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Working Group Chairs <wgchairs.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/wgchairs>, <mailto:wgchairs-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/wgchairs/>
List-Post: <mailto:wgchairs@ietf.org>
List-Help: <mailto:wgchairs-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/wgchairs>, <mailto:wgchairs-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 20 Aug 2020 01:42:54 -0000

> On 20 Aug 2020, at 11:33 am, Bron Gondwana <brong@fastmailteam.com> wrote:
> 
> I'm more interested in the IETF maintaining a repository of working test cases and test suites.  Test suites are much more valuable to new people coming along trying to implement the spec, and even just good examples and test cases can help development significantly, but don't necessarily have to be spelled out inside the RFC because they take up lots of space and the RFC formatting is not conducive to things like precise whitespace and line endings!

There are a number of ways that this already happens.

E.g., the QUIC WG uses a Slack for implementers to discuss interop, and a spreadsheet to track status:
  https://docs.google.com/spreadsheets/d/1D0tW89vOoaScs3IY9RGC0UesWGAwE6xyLk0l4JtvTVg/edit#gid=1268516408

The HTTP WG has a list of unofficial testing resources:
  https://github.com/httpwg/wiki/wiki/HTTP-Testing-Resources
... and also hosts some tests in its GitHub org:
  https://github.com/httpwg/structured-field-tests

This can also be done by the community, with help from WG participants and authors:
  https://github.com/json-patch/json-patch-tests

I think the relevant question is how 'official' this needs to be. The big issue that usually comes up is that official tests implies a testing /conformance regime, which is expensive to run properly, and can be legally risky for a standards body. Keeping things a little bit unofficial helps to avoid this, while still encouraging interoperability.

I think all of this is in-scope for the EDM program; if you're interested, see:
  https://www.iab.org/activities/programs/evolvability-deployability-maintainability-edm-program/

Cheers,


> Bron.
> 
> On Thu, Aug 20, 2020, at 11:30, Mark Nottingham wrote:
>> Hi Vijay,
>> 
>> Just my .02 - I'd be wary of an IETF repository of such code, as that would convey too much authority on it. It would also create a fair amount of overhead (especially for long-term maintenance).
>> 
>> I also don't think that it's necessary, once a protocol is successful; after broad deployment, people don't need to go searching for implementations. It *is* useful when a project is starting, to help early implementers find each other, though. 
>> 
>> In the WGs I participate in, we often use wiki pages to track early implementations; e.g.:
>> 
>> * https://github.com/httpwg/wiki/wiki/Structured-Headers
>> * https://github.com/quicwg/base-drafts/wiki/Implementations
>> 
>> Cheers,
>> 
>> 
>> > On 20 Aug 2020, at 1:10 am, Vijay Gurbani <vijay.gurbani@gmail.com> wrote:
>> > 
>> > All: I was reviewing a draft [1] as part of Gen-ART.  In the draft, there is a section pertaining to "Implementation Status" as dictated by RFC 7942..  I found this section to be very interesting in that it captures the running code corresponding to the I-D, but RFC 7942 does not appear to provide the means to retain this section upon publication as a RFC.  (To be sure, it does not mandate that the section be removed, just suggests it.  However, most I-D authors simply will end up asking the RFC editor to remove the section, as [1] does.)
>> > 
>> > I think that is rather counter productive.  After all, we standardize protocols so that others can write programs that implement the protocols, and I see a lot of value in preserving any running code.  In the particular case of the I-D I reviewed, there were two implementations, both from reputable organizations (APNIC and the Italian National Research Council).  By simply deleting the "Implementation Status" section when the I-D was published as an RFC, it seems that good, quality implementations that folks spent time on would be lost, perhaps not irrevocably, but for most practical purposes, the code would be orphaned.
>> > 
>> > I am wondering whether the IETF should preserve some of this running code as an archive.  The IETF makes no guarantees of the correctness of the code, just provides a repository where code can be linked to a particular RFC to allow implementers to bootstrap themselves, if needed.  This repository will also contain the contact of the original author of the code.  The repository is sort of like an IANA registry of IETF working code corresponding to an RFC.  The authors of the I-D can be the "expert" and suggest which implementations they consider worthy of archival.  The boilerplate of implementations is given in Section 2 of RFC 7942 [2].  We would have to spec out an IANA-registry like process to do so, but I think that the ends will justify the means.
>> > 
>> > I note that organizations like IEEE have evolved to provide a repository where authors of published papers can park their datasets for reproducibility (see IEEE DataPort [3]).  The 2017 ACM Task Force on Data, Software, and Reproducibility in Publication also recommended that, "The ACM Digital Library will need to interoperate with a number of recommended data repositories and software curation platforms..." [4]..  So this idea of the IETF maintaining a software repository is not entirely unheard of.
>> > 
>> > I apologize for taking your time if this has been discussed before; if not, it is worth some discussion.
>> > 
>> > [1] https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-15
>> > [2] https://tools.ietf.org/html/rfc7942
>> > [3] https://bigdata.ieee.org/ieee-dataport
>> > [4] https://www.acm.org/publications/task-force-on-data-software-and-reproducibility
>> > 
>> > Thank you,
>> > 
>> > - vijay
>> 
>> --
>> Mark Nottingham   https://www.mnot.net/
>> 
>> 
> 
> --
>   Bron Gondwana, CEO, Fastmail Pty Ltd
>   brong@fastmailteam.com

--
Mark Nottingham   https://www.mnot.net/