Re: [spring] Proposed policy on reporting implementation and interoperability

[Joel Halpern <[e-mail]>]

Thank you for your comments.  I would like to see more discussion of 
your in line points by the working group.   For your initial questions 
my personal comments are in line.

Yours,

Joel

On 8/19/2022 12:28 AM, Ketan Talaulikar wrote:
> Hi Joel/All,
>
> Can the policy clarify some of the following points which are not 
> explicitly covered in its currently proposed text?
>
> a) Whether a single implementation is sufficient or if we require at 
> least 2 *independent* (i.e., by different implementors) ones?

Unlike IDR, we are not requiring any implementations.  I hope we will 
hear about multiple independent implementations, but we are not 
proposing to make that a requirement for publication. Somewhat different 
rules make sense in different working groups.


> b) There are some MUSTs that are associated with optional features 
> (i.e., the MUST come into play only when someone is implementing that 
> specific option). Should there not be a distinction between the 
> basic/mandatory and optional MUSTs?

I would be interested in hearing from the WG on this.  My expectations 
is that if someone says they implement optional feature X, and X has 
MUSTs conditioned on it, then they have to explain whether they comply 
with those MUSTs.  (I would hope they do so.  the WG would hope they do 
so.  But we are not the protocol police.)


> c) SHOULDs fall between mandatory and recommendations. In most cases, 
> they are quite important/critical and at almost the same level as the 
> MUSTs when it comes to functionality. Should they not be covered 
> similarly to the MUSTs?

Interesting point.  Do others agree that the policy should include 
requiring explanations of SHOULDs?


>
> Please find inline below some observations/comments/feedback on this 
> proposed policy.
>
>
> On Wed, Aug 3, 2022 at 8:15 PM Joel Halpern <[e-mail]> wrote:
>
>     SPRING WG:
>
>     At the suggestion of our AD, the WG Chairs have been discussing
>     whether it would be helpful to be more explicit, in I-Ds and RFCs
>     we produce, about the announced implementations and known
>     interoperability tests that have occurred.
>
> KT> Looking at the SPRING WG charter, the work done here is mostly 
> architectural in nature and the necessary protocol extensions to 
> actually deliver the feature(s) are done in other WGs. Does it not 
> make sense to extend or apply this policy across WGs (at least those 
> in the RTG area that actually manifest SR solutions)? Without that, I 
> question if such a policy's application to SPRING WG alone is 
> effective. Has there been any discussion on this policy amongst the 
> RTG ADs and other RTG area WG chairs? If so, would be good to get some 
> insights into the wider thought process (if there is one) from our ADs.
>
>     If the WG agrees, we would like to institute and post on the WG
>     wiki the following policy.  The period for discussion and comment
>     runs until 9-Sept-2022, to allow for folks who are on summer break:
>
>     All I-Ds that reach WG last call shall have an implementation
>     section based on, but somewhat more than, that described in RFC
>     7942 (BCP 205,*Improving Awareness of Running Code: The
>     Implementation Status Section*). Authors are asked to collect
>     information about implementations and include what they can find
>     out when that information is available for public disclosure.
>     Documents will not be blocked from publication if the authors fill
>     in the section as "none report" when they have made an effort to
>     get information and not been able to.
>
> KT> The proof of the pudding is not in its recipe but in actually 
> making and then eating it :-) Therefore, I strongly support the 
> inclusion of an Implementation Section in the WG documents per 
> RFC7942. It is a very good idea to mandate this as part of SPRING WG 
> policy.
>
>     There are a couple of important additions to what is called for in
>     RFC 7942.  We have confirmed with leadership that these changes
>     are acceptable in terms of IETF process:
>
>     1) We will retain the implementation status section when the draft
>     is published as an RFC.  In order to do so, the section will begin
>     with "this is the implementation status as reported to the
>     document editors as of <date>"
>
> KT> I am strongly opposed to this since such "transient" and 
> "changing" data (even with a date stamped) is unsuitable for 
> publication in the form of an RFC. Instead, I recommend that the WG 
> follow https://www.rfc-editor.org/rfc/rfc7942.html#section-3. We can 
> pick any/all of the options and make them available for consideration.
>
> KT> Another aspect to consider is that publishing these details in the 
> RFCs may give an advantage to some vendors who are the first to 
> implement (for whatever reasons) and hence get mentioned in the RFC 
> while some other vendor that implements the feature post-publication 
> has no chance to get on the RFC. This might be seen as incentivizing 
> vendors that are in a position to implement first.
>
>     2) Each implementation description MUST include either a statement
>     that all MUST clauses in the draft / RFC are implemented, or a
>     statement as to which ones are not implemented.
>
> KT> As someone who has been an author/editor of IDR WG documents, I 
> can share that getting this level of information has been a rather 
> tedious and challenging process. It may be a good target to set out - 
> though I am not sure about making this mandatory. My concern is that 
> we might not always be able to get this level of details and then the 
> question will arise before the WG on how long to delay the 
> publication. Note that we do architecture work in SPRING and the 
> protocol extensions are done in other WGs. What happens when those 
> other WGs don't have a similar/consistent policy and the protocol 
> specs that are dependent on the SPRING WG architecture doc get stuck 
> awaiting implementation report details?
>
>     3) each implementation description may include reports of what
>     optional elements of the draft / RFC are implemented.
>
> KT> Same as the previous comment.
>
>     Reports of interoperabiity testing are strongly encouraged. 
>     Including the reports in the document is preferred.  This may
>     include a reference to longer and more detailed testing reports
>     available elsewhere.  If there are no reports of interoperability
>     tests, then the section MUST state that no such reports were received.
>
> KT> Capturing pointers to or details of interoperability testing in 
> the implementation report is a good idea. Subject to my response to 
> (1) above. Interoperability testing at forums like the EANTC is an 
> ongoing (yearly) process. More/new vendors get added as the feature 
> matures and gets wider adoption. Putting this into an RFC publication 
> does not make sense.
>
> Thanks,
> Ketan
>
>     Yours,
>
>     Bruno, Jim, and Joel
>
>     _______________________________________________
>     spring mailing list
>     [e-mail]
>     https://www.ietf.org/mailman/listinfo/spring
>