Re: [DNSOP] raising the bar: requiring implementations

[Willem Toorop <[e-mail]>]

I would love to see a hard requirement for implementations &
implementation reports (like IDR has) in the charter or in the working
group house rules.  Early implementations (perhaps even during the
hackathon) can reveal implications that might have been missed while
designing the draft.  In addition it is a good way to see if everybody
interprets a draft the same way, by interoperability testing.


Op 24-03-18 om 12:07 schreef Job Snijders:
> Dear DNSOP,
> 
> I'd like to share some pointers from the working group that governs the
> BGP protocol, IDR, on requiring implementations before drafts can
> advance towards RFC publication. Raising the bar for publication will
> take weight off the camel's back.
> 
> The IDR policy and rationale is outlined here: https://trac.ietf.org/trac/idr/wiki
> 
> An example of an implementation report is available here:
> https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-large-community%20implementations
> Every normative (RFC 2119 / RFC 8174) term should expand into a
> compliance test, and it is particularly good to document where
> implementations deviate from what is required or suggested in the
> proposed specification. The work listed on this wikipage was done
> _before_ the draft was submitted to the IESG, and we intentionally
> sought to create more than the minimum amount of implementations
> required.
> 
> In the case of the BGP large communities project the working group was
> particularly lucky because Pier Carlo Chiodi created an open 'regression
> testing'-style environment into which BGP speakers could be plugged in
> to assess their compliance:
> https://github.com/pierky/bgp-large-communities-playground
> https://github.com/pierky/bgp-large-communities-playground/blob/master/tests/README.md
> 
> The DNS world would benefit from such environments and automated
> compliance testing. Manual testing for a specification (that is still
> being worked on) can be tedious, having a 'playground' can pay huge
> dividend. We did catch bugs in implementations thanks to this
> environment, so it not only helped the specification, but increased the
> quality of the participating implementations.
> 
> RFC 7942 suggest that During the development of a draft people are
> encouraged to document what implementations exist, an example is here:
> https://tools.ietf.org/html/draft-ietf-idr-large-community-12#section-7
> 
> Closed source implementations as part of such a list is not an issue
> (just a little bit more work), at IETF meetings we'd meet up, plug
> laptops with virtual machines into each other and run compliance tests.
> Sidenote: when IANA codepoints are needed but not yet assigned, don't
> forget to keep everything on separate beta/feature branches; don't ship
> software with squatted codepoints :-)
> 
> The DNSOP working group will have to figure out what constitutes a
> meaningful implementation in what context. For example, a tcpdump or
> wireshark decoder would hardly count as an implementation of a proposed
> DNS feature, but those decoders are _incredibly_ useful to have
> alongside real implementations, especially during development. DNS
> people will probably want to see multiple implementation reports in
> context of packet decoding, stub, resolver, and authoritative.
> 
> Kind regards,
> 
> Job
> 
> _______________________________________________
> DNSOP mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dnsop
>