draft-farrell-ft-01.txt -- what signal are we attempting to sense?

[Ted Hardie <[e-mail]>]

Reading through Stephen's draft and the discussion to date, I think there
is some confusion/disagreement about what it is having an implementation
at this stage signals.

One way to break up the work of the IETF is:

Engineering--making decisions about the trade-offs related to
             different approaches to solving a problem.

Specification--producing text that describes how to inter-operate with
others.

Standardization--describing the applicability of a specification or
         its suitability as the basis of other work

(Since we reflect all of these in the same document production, it's
really muddier than this, but bear with me)

My experience is that people implementing during the working group
discussion phase generate really useful data about the engineering;
they can tell you the real impact of different trade-offs, so that
this isn't based on general experience.  But it's not such a great
signal about the specification itself, since the spec is designed to
be usable by folks who were not part of the working group process.

Stephen's draft says:

   Note also that this experiment just needs an implementation that
   makes it possible for the WG chairs and responsible AD to verify (to
   the extent they chose) that the implementation matches the draft.

and later:

   An implementation of the draft (ideally open-source) is required
   for fast-track last-call.  If there is no implementation or if the
   implementation is unavailable or does not implement the draft
   sufficiently closely then the document needs to be returned to the
   WG.  This only requires one implementation, not two and the WG
   chairs and responsible AD decide themselves how much validation is
   required for this.

Given the "sufficiently closely" and the timing of production, I
assume that the signal we're looking for here is confirmation of the
engineering choices.  I think that's fine (though I'm not sure this
needs formal experiment status).  But I believe we need to be really
careful that it isn't mistaken for signal about the specification's
quality.  It can happen that a working group has "lore" about what to
do that gets folded into the implementations done by those
participating, but which never quite makes it into the spec "because
everyone knows it".  An implementation written during the working
group process is potentially subject to this effect.

An interoperating implementation written to the spec by a non-working
group participant would be great signal about the specification quality,
but it  is not likely to be available at the stage of the process this
draft
targets.

Again, not objecting to the experiment; I just want to be clear about
what signal we believe we're getting from the implementation.

Just my two cents,

Ted