Re: [DNSOP] raising the bar: requiring implementations

[Petr Špaček <[e-mail]>]

On 24.3.2018 12:07, Job Snijders wrote:
> Dear DNSOP,
> 
> I'd like to share some pointers from the working group that governs the
> BGP protocol, IDR, on requiring implementations before drafts can
> advance towards RFC publication. Raising the bar for publication will
> take weight off the camel's back.
> 
> The IDR policy and rationale is outlined here: https://trac.ietf.org/trac/idr/wiki
> 
> An example of an implementation report is available here:
> https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-large-community%20implementations
> Every normative (RFC 2119 / RFC 8174) term should expand into a
> compliance test, and it is particularly good to document where
> implementations deviate from what is required or suggested in the
> proposed specification. The work listed on this wikipage was done
> _before_ the draft was submitted to the IESG, and we intentionally
> sought to create more than the minimum amount of implementations
> required.

I very much support this approach.

BTW we already have test framework which was used to test BIND, Unbound,
Knot Resolver, and PowerDNS recursor! I.e. we have one tool which is
able to test all of these using the same test data, so the remaining
work here is writting tests, not creating test tools from scratch.

Petr Špaček  @  CZ.NIC


> In the case of the BGP large communities project the working group was
> particularly lucky because Pier Carlo Chiodi created an open 'regression
> testing'-style environment into which BGP speakers could be plugged in
> to assess their compliance:
> https://github.com/pierky/bgp-large-communities-playground
> https://github.com/pierky/bgp-large-communities-playground/blob/master/tests/README.md
> 
> The DNS world would benefit from such environments and automated
> compliance testing. Manual testing for a specification (that is still
> being worked on) can be tedious, having a 'playground' can pay huge
> dividend. We did catch bugs in implementations thanks to this
> environment, so it not only helped the specification, but increased the
> quality of the participating implementations.
> 
> RFC 7942 suggest that During the development of a draft people are
> encouraged to document what implementations exist, an example is here:
> https://tools.ietf.org/html/draft-ietf-idr-large-community-12#section-7
> 
> Closed source implementations as part of such a list is not an issue
> (just a little bit more work), at IETF meetings we'd meet up, plug
> laptops with virtual machines into each other and run compliance tests.
> Sidenote: when IANA codepoints are needed but not yet assigned, don't
> forget to keep everything on separate beta/feature branches; don't ship
> software with squatted codepoints :-)
> 
> The DNSOP working group will have to figure out what constitutes a
> meaningful implementation in what context. For example, a tcpdump or
> wireshark decoder would hardly count as an implementation of a proposed
> DNS feature, but those decoders are _incredibly_ useful to have
> alongside real implementations, especially during development. DNS
> people will probably want to see multiple implementation reports in
> context of packet decoding, stub, resolver, and authoritative.