Re: [Sidrops] interpretation of consensus on requiring running code for SIDROPS standards track docs

[Andrew Newton <[e-mail]>]

Hi Job,

Sorry for being ambiguous. I'm generally agreeable to your proposal.

:)

-andy

On Wed, Nov 8, 2023 at 3:09 PM Job Snijders
<job=[e-mail]> wrote:
>
> Dear all,
>
> Following conversations onlist and offlist, I propose to add the
> following text to the SIDROPS charter:
>
>     "Before SIDROPS Standards Track internet-drafts can progress to IESG
>      review, interoperability must be demonstrated for two independently
>      developed implementations for all aspect of the concepts in the
>      specification. The SIDROPS chairs can waive this requirement at
>      their discretion."
>
> I'd like to share my personal interpretation of the working group
> consensus on this topic. I've analysed the two main mail threads on this
> topic and quoted or summarized my interpretation of the position of each
> participant.
>
> This overview is intended to help the chairs to drive towards a
> conclusion. I do not intend to put words in people's mouths; my
> apologies if I mischaracterized anyone's message.
>
> December 2020 thread https://mailarchive.ietf.org/arch/msg/sidrops/6o9t4m9vBCEbpVVtDSQHjLyURNY/
> July 2023 thread https://mailarchive.ietf.org/arch/msg/sidrops/SMoaxTRho2A3AH2zJEouoRrQka8/
>
> Agreeing:
> =========
>
> Nick Hilliard: "this does not seem to be a completely dumb idea"
> https://mailarchive.ietf.org/arch/msg/sidrops/66Zeu3iWXnSMGkPQws9yYaj33I4/
>
> Lukas Tribus: "I support this proposal."
> https://mailarchive.ietf.org/arch/msg/sidrops/hM_T8bG7xhnve3bQpH0yzS3C72M/
>
> Jakob Heitz: "I support."
> https://mailarchive.ietf.org/arch/msg/sidrops/knVQaKPDY7Gt0_5grFqUvFDv_J4/
>
> Joel Jaeggli: "If we take on the protocol specification maintenance here
>                there absolutely we should require interoperable
>                implementation for standards advancement."
> https://mailarchive.ietf.org/arch/msg/sidrops/m7ZxDj10HZnagx95sZ1ZkMh_Akw/
>
> Tim Bruijnzeels: (paraphrased) "I believe that it would be good to
>                 require two implementations for standards track." and
>                 also "In principle, I agree with the proposal"
> https://mailarchive.ietf.org/arch/msg/sidrops/gJTKUgqpWrZdb3FtONp5RJytVtY/
> https://mailarchive.ietf.org/arch/msg/sidrops/-mF0HwgCVsu1g33ldpOrlWK1dbw/
>
> Jay Borkenhagen: "I support. I believe this change will help improve
>                  the quality of RFCs produced here by requiring thorough
>                  and detailed understanding by implementers before the
>                  drafts become RFCs."
> https://mailarchive.ietf.org/arch/msg/sidrops/1YB8NzGGElGIJCztbIsc9IPjxMU/
>
> George Michaelson: "I think this would be sensible. From a CA,
>                    producer-side point of view, I think having
>                    confidence of interop demonstrated, and having this
>                    in both producer and consumer (signer and validator)
>                    would be net beneficial."
> https://mailarchive.ietf.org/arch/msg/sidrops/Oal41iVkPT1cnAgnz3eyt4XgRa0/
>
> Christopher Morrow: "I think these things sound good to me (need
>                      implementations before we release the kraken(s) )."
>                      (Note: Chris is SIDROPS co-chair.)
> https://mailarchive.ietf.org/arch/msg/sidrops/p2Nm9K0M6JLqjNghgD5ktbVC3VY/
>
> Ben Maddison: "I agree, on a balance, that running code should be a
>                pre-requisite for standards track documents."
> https://mailarchive.ietf.org/arch/msg/sidrops/Nqn2lWKT5YIEbs98hIA5CL-n9Qs/
>
> Dale W. Carder: (paraphrased) "Section 4 of RFC 7942 lists advantages of
>                                running code, these outweigh the
>                                potential downsides"
> https://mailarchive.ietf.org/arch/msg/sidrops/m3DZWu_e5bmXoBIxD7ZPPF0MuT8/
>
> Fredrik Korsbaeck: "it seems like a sensible development to require
>                     running code"
> https://mailarchive.ietf.org/arch/msg/sidrops/3ahRPP5nz4nE4e7Ucqu0JaprZWQ/
>
> Job Snijders: "YES, REQUIRE RUNNING CODE"
> https://mailarchive.ietf.org/arch/msg/sidrops/CKt63-mBQ1omN5HiE1_o1576Kag/
>
> Nuance
> ======
>
> While quite some people expressed unambiguous support, there were also a
> few that suggested there is room for nuance:
>
> Geoff Huston: (paraphrased) "I question the wisdom of introducing a
>                             _strict_ requirement", and also: "feedback
>                             from early implementations can assist in the
>                             quality of a specification."
> https://mailarchive.ietf.org/arch/msg/sidrops/fFs0HoREZ5dvSu5R-f7iIP0pOqU/
>
> Andrew Newton: "might be wise to consider an escape hatch for things
>                 that might be more urgent but would otherwise get hung
>                 up on a formal interoperability report." and also
>                 (paraphrased) "how IDR does it looks good to me"
> https://mailarchive.ietf.org/arch/msg/sidrops/n6b7rCSMOJthSSvSdNpc6gvNrqk/
> https://mailarchive.ietf.org/arch/msg/sidrops/VToDXcZuLNw5RQaKwFZJ4IbbZE8/
>
> To address the above, I added the sentence that the chairs can just
> waive the requirement at their discretion.
>
> Dissent
> =======
>
> I did not see explicitly dissenting opinions on the matter or requiring
> running code; but I am aware of some (hypothetical) counter-arguments:
>
> A) Some organisations prefer to only implement things that have achieved
>    published RFC status. This could introduce deadlocks in the working
>    group's progression.
> B) Requiring running code increases the burden on internet-draft
>    authors, as they might might not have the resources to instrument
>    implementations of their proposal.
> C) Internet-drafts are a moving target, preference to only have to
>    implement things once and not have to re-visit the work often.
>
> On the whole I think each of these counter-arguments relate to sides of
> the same coin: (A) organisations are free to prioritize and schedule
> their implementation work as they deem appropriate, some organisations
> do not wish to participate in the development of open standards, on the
> other hand I believe there always will be orgs that *do* want to engage
> and invest prior to RFC publication. (B) Authors not having the
> resources to produce implementations will encourage those authors to
> team up with implementers. I think this is healthy and fosters
> collaboration. (C) authors will always have to be considerate when
> introducing new breaking changes in new versions of a draft, being
> cognizant of the burden we impose on others is part of the process.
>
> Conclusion
> ==========
>
> By and large I see broad support for requiring running code in the form
> of implementation reports before progressing internet-drafts beyond
> WGLC. As there is no predefined form what exactly constitutes an
> implementation report, I believe we can rely on the chairs' judgement.
>
> Authors of Standards track SIDROPS internet-drafts would do well to
> always include a RFC 7942 section in their draft.
>
> The SIDROPS chairs should request a 'SIDROPS wiki'
> https://wiki.ietf.org/group/sidrops to be created; so that people can
> easily enter and update interopability matrix tables.
>
> Thank you to everyone for chiming in on this important question.
>
> Kind regards,
>
> Job
>
> _______________________________________________
> Sidrops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sidrops