Re: [spring] Proposed policy on reporting implementation and interoperability

Joel Halpern <jmh@joelhalpern.com> Fri, 12 August 2022 14:52 UTC

Return-Path: <jmh@joelhalpern.com>
X-Original-To: spring@ietfa.amsl.com
Delivered-To: spring@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C8E8EC15A728 for <spring@ietfa.amsl.com>; Fri, 12 Aug 2022 07:52:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.126
X-Spam-Level:
X-Spam-Status: No, score=-2.126 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=joelhalpern.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XXZN0si-Wv4A for <spring@ietfa.amsl.com>; Fri, 12 Aug 2022 07:52:29 -0700 (PDT)
Received: from mailb2.tigertech.net (mailb2.tigertech.net [208.80.4.154]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E1191C157B4B for <spring@ietf.org>; Fri, 12 Aug 2022 07:52:29 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by mailb2.tigertech.net (Postfix) with ESMTP id 4M468s3tF7z1pY3x; Fri, 12 Aug 2022 07:52:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelhalpern.com; s=2.tigertech; t=1660315949; bh=wMNtBe+2XqEE+8xgKjXoRPzAlAOqoVeBXlJFXrSN8Zc=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=Xpsk3sseX2W/QAQK3nW+K7vTPdFKWUwscGfvfEJA1xBzGcXXk4oERxTwFj61WY+2i i+cUCfUe5bhJPVdC9Eh3diJ1noB1VVbITp2BLXaPDN75cDwX8A8eFTPzvyviIWw9kI gt3mFclRuUjbboiiao31+OzjLEFYC5pYqQIarUC0=
X-Quarantine-ID: <VKQj6GTm2dFh>
X-Virus-Scanned: Debian amavisd-new at b2.tigertech.net
Received: from [192.168.23.181] (unknown [50.233.136.230]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mailb2.tigertech.net (Postfix) with ESMTPSA id 4M468q5lYpz1pY3k; Fri, 12 Aug 2022 07:52:27 -0700 (PDT)
Content-Type: multipart/alternative; boundary="------------8TqD8BkVdkkzjbiw5iq21VCd"
Message-ID: <cdf901a3-117b-051c-d410-aace8ed11600@joelhalpern.com>
Date: Fri, 12 Aug 2022 10:52:26 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0
Content-Language: en-US
To: "Eric Vyncke (evyncke)" <evyncke@cisco.com>, "spring@ietf.org" <spring@ietf.org>
Cc: Andrew Alston <Andrew.Alston@liquidtelecom.com>, John Scudder <jgs@juniper.net>, Alvaro Retana <aretana.ietf@gmail.com>
References: <9c7ac280-c1f7-956c-cdbb-2b0745aaf2fa@joelhalpern.com> <3EABB4EF-CA37-44A3-8DF6-0029C1E53C5B@cisco.com>
From: Joel Halpern <jmh@joelhalpern.com>
In-Reply-To: <3EABB4EF-CA37-44A3-8DF6-0029C1E53C5B@cisco.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/spring/P8FUVyKu7gD_wRWI_2Qa-Wz5Sj0>
Subject: Re: [spring] Proposed policy on reporting implementation and interoperability
X-BeenThere: spring@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: "Source Packet Routing in NetworkinG \(SPRING\)" <spring.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/spring>, <mailto:spring-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/spring/>
List-Post: <mailto:spring@ietf.org>
List-Help: <mailto:spring-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/spring>, <mailto:spring-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Aug 2022 14:52:33 -0000

I assume that if you have these questions, so do other people. So even 
though you said you do not seek a reply, it seems useful to the 
community to provide some form of reply.  I have not consulted with my 
co-chairs, so these are my personal understandings of what this means.  
If there is wording clarification that people would like please someone 
suggest such. (I believe we have wording to address Dhruv's comment.)  
The final policy will end up on the WG wiki.

My responses to your questions are in line.   Marked <jmh> </jmh> so 
that boundaries are clear when mailers mangle this.

Yours,

Joel

On 8/11/2022 9:57 AM, Eric Vyncke (evyncke) wrote:
>
> Bruno, Jim, Joel,
>
> Without any hat, or only with the experience of reviewing so many I-Ds 
> from different areas and having implemented a couple of 
> proof-of-concept implementations of various IETF drafts.
>
> The following are just comments for discussion, not criticisms because 
> indeed running code is important at the IETF.
>
> RFC 7942, an IETF stream BCP (i.e., with IETF consensus), is already 
> useful but **optional** while the proposal below would be mandatory. 
> Hence, I share Dhruv’s question: will it also apply to experimental or 
> informational documents or BCP? Joel, thank you for your reply to 
> Dhruv, it seems a sensible solution.
>
<jmh>I believe the second part of the above is addressed.  To be clear 
about the first part, yes, we are mandating that the document editors 
make an effort to find out about implementations and interoperability 
tests, and include that in the document.  Or mark that they were unable 
to find any. (Or, as noted, mark that the need does not apply to this 
document.)  Also as noted, we are keeping this in the RFC.  These 
actions are permitted by the BCP, although the BCP does not go as far as 
this. </jmh>
>
> You may be aware of a long-standing project/idea/wish of the IESG to 
> have a **living document concept** that could be updated for years 
> after RFC publication. As you can imagine there are too many gears to 
> make fast progress on this topic, but the current proposal is to move 
> the ‘living’ part of the doc to a github repo under the IETF 
> organization (e.g., done in MOPS). Wouldn’t it be a better fit than a 
> section ‘carved in stone’ forever (even if time-stamped) in an RFC?
>
<jmh>A living document recording implementation and interoperability 
would be a different thing.  The value here is specifically in capturing 
the state as it was known to the community at the time of submitting the 
document for RFC publication.  If people want to create and maintain 
wiki pages in the working group wiki about ongoing activity, that would 
be great.  If folks step forward we will have to figure out with the 
tools team how to support that.  However, we are not asking folks to do 
that. Particularly since for them to be of value someone has to perform 
ongoing maintenance of the report.  </jmh>
>
> Getting an implementation could sometimes be easy in Python or Go in 
> Linux at a hackathon  ;-), or in “slow path”, a little more complex in 
> P4, even more in silicon... the ultimate step is of course a 
> **deployment** outside of a virtualized lab:, i.e., in a production 
> network. Of course, deployment (and interoperation) in a production 
> network is the Graal ;-) But, where to put a useful but realistic 
> needle **without delaying** the standardization process ?
>
<jmh>As far as I know, we are not engaging in discussion of the quality 
or value of the implementation.  Just its existence and features.  And 
then whether we have reports of its interoperability.  From a process 
perspective I can not imagine us getting involved in judging which 
implementations are or are not relevant.</jmh>
>
> Finally, while I understand that having a detailed (up to the MUST 
> level) description could be useful, this may also quickly change over 
> time and will probably require too much of work by the I-D authors and 
> implementers. The IETF should also be very careful in the wording of 
> this section as the IETF is not a compliance testing lab, i.e., it is 
> not because a published RFC claims that vendor foo has implemented 
> protocol bar that it has really implemented protocol bar.
>
<jmh>If you do not feel the proposed wording is clear enough about the 
information being a snapshot in time, please suggest further wording. </jmh>
>
> Again, written w/o any hat and hoping to help a useful discussion
>
> Regards
>
> -éric
>
> *From: *spring <spring-bounces@ietf.org> on behalf of Joel Halpern 
> <jmh@joelhalpern.com>
> *Date: *Wednesday, 3 August 2022 at 16:56
> *To: *SPRING WG List <spring@ietf.org>
> *Subject: *[spring] Proposed policy on reporting implementation and 
> interoperability
>
> SPRING WG:
>
> At the suggestion of our AD, the WG Chairs have been discussing 
> whether it would be helpful to be more explicit, in I-Ds and RFCs we 
> produce, about the announced implementations and known 
> interoperability tests that have occurred.  If the WG agrees, we would 
> like to institute and post on the WG wiki the following policy.  The 
> period for discussion and comment runs until 9-Sept-2022, to allow for 
> folks who are on summer break:
>
> All I-Ds that reach WG last call shall have an implementation section 
> based on, but somewhat more than, that described in RFC 7942 (BCP 
> 205,*Improving Awareness of Running Code: The Implementation Status 
> Section*).  Authors are asked to collect information about 
> implementations and include what they can find out when that 
> information is available for public disclosure.  Documents will not be 
> blocked from publication if the authors fill in the section as "none 
> report" when they have made an effort to get information and not been 
> able to.
>
> There are a couple of important additions to what is called for in RFC 
> 7942.  We have confirmed with leadership that these changes are 
> acceptable in terms of IETF process:
>
> 1) We will retain the implementation status section when the draft is 
> published as an RFC.  In order to do so, the section will begin with 
> "this is the implementation status as reported to the document editors 
> as of <date>"
>
> 2) Each implementation description MUST include either a statement 
> that all MUST clauses in the draft / RFC are implemented, or a 
> statement as to which ones are not implemented.
>
> 3) each implementation description may include reports of what 
> optional elements of the draft / RFC are implemented.
>
> Reports of interoperabiity testing are strongly encouraged. Including 
> the reports in the document is preferred.  This may include a 
> reference to longer and more detailed testing reports available 
> elsewhere.  If there are no reports of interoperability tests, then 
> the section MUST state that no such reports were received.
>
> Yours,
>
> Bruno, Jim, and Joel
>