[DNSOP] raising the bar: requiring implementations

[Job Snijders <[e-mail]>]

Dear DNSOP,

I'd like to share some pointers from the working group that governs the
BGP protocol, IDR, on requiring implementations before drafts can
advance towards RFC publication. Raising the bar for publication will
take weight off the camel's back.

The IDR policy and rationale is outlined here: https://trac.ietf.org/trac/idr/wiki

An example of an implementation report is available here:
https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-large-community%20implementations
Every normative (RFC 2119 / RFC 8174) term should expand into a
compliance test, and it is particularly good to document where
implementations deviate from what is required or suggested in the
proposed specification. The work listed on this wikipage was done
_before_ the draft was submitted to the IESG, and we intentionally
sought to create more than the minimum amount of implementations
required.

In the case of the BGP large communities project the working group was
particularly lucky because Pier Carlo Chiodi created an open 'regression
testing'-style environment into which BGP speakers could be plugged in
to assess their compliance:
https://github.com/pierky/bgp-large-communities-playground
https://github.com/pierky/bgp-large-communities-playground/blob/master/tests/README.md

The DNS world would benefit from such environments and automated
compliance testing. Manual testing for a specification (that is still
being worked on) can be tedious, having a 'playground' can pay huge
dividend. We did catch bugs in implementations thanks to this
environment, so it not only helped the specification, but increased the
quality of the participating implementations.

RFC 7942 suggest that During the development of a draft people are
encouraged to document what implementations exist, an example is here:
https://tools.ietf.org/html/draft-ietf-idr-large-community-12#section-7

Closed source implementations as part of such a list is not an issue
(just a little bit more work), at IETF meetings we'd meet up, plug
laptops with virtual machines into each other and run compliance tests.
Sidenote: when IANA codepoints are needed but not yet assigned, don't
forget to keep everything on separate beta/feature branches; don't ship
software with squatted codepoints :-)

The DNSOP working group will have to figure out what constitutes a
meaningful implementation in what context. For example, a tcpdump or
wireshark decoder would hardly count as an implementation of a proposed
DNS feature, but those decoders are _incredibly_ useful to have
alongside real implementations, especially during development. DNS
people will probably want to see multiple implementation reports in
context of packet decoding, stub, resolver, and authoritative.

Kind regards,

Job