Re: draft-farrell-ft-01.txt -- what signal are we attempting to sense?

[Stephen Farrell <[e-mail]>]

Hi Ted,

Thanks that change looks good to me. I'll whack it in thanks. I do
still like the word "reward" though so I'll tag that on too:-)

Cheers,
S.


On 12/05/2012 07:36 PM, Ted Hardie wrote:
> Hi Stephen,
> 
> Some further comments in-line.
> 
> On Wed, Dec 5, 2012 at 9:58 AM, Stephen Farrell
> <[e-mail]>wrote:
> 
>>
>> Hi Ted,
>>
>> On 12/05/2012 05:22 PM, Ted Hardie wrote:
>>> Reading through Stephen's draft and the discussion to date, I think there
>>> is some confusion/disagreement about what it is having an implementation
>>> at this stage signals.
>>>
>>> One way to break up the work of the IETF is:
>>>
>>> Engineering--making decisions about the trade-offs related to
>>>              different approaches to solving a problem.
>>>
>>> Specification--producing text that describes how to inter-operate with
>>> others.
>>>
>>> Standardization--describing the applicability of a specification or
>>>          its suitability as the basis of other work
>>>
>>> (Since we reflect all of these in the same document production, it's
>>> really muddier than this, but bear with me)
>>>
>>> My experience is that people implementing during the working group
>>> discussion phase generate really useful data about the engineering;
>>> they can tell you the real impact of different trade-offs, so that
>>> this isn't based on general experience.  But it's not such a great
>>> signal about the specification itself, since the spec is designed to
>>> be usable by folks who were not part of the working group process.
>>
>> Good point.
>>
>>> Stephen's draft says:
>>>
>>>    Note also that this experiment just needs an implementation that
>>>    makes it possible for the WG chairs and responsible AD to verify (to
>>>    the extent they chose) that the implementation matches the draft.
>>>
>>> and later:
>>>
>>>    An implementation of the draft (ideally open-source) is required
>>>    for fast-track last-call.  If there is no implementation or if the
>>>    implementation is unavailable or does not implement the draft
>>>    sufficiently closely then the document needs to be returned to the
>>>    WG.  This only requires one implementation, not two and the WG
>>>    chairs and responsible AD decide themselves how much validation is
>>>    required for this.
>>>
>>> Given the "sufficiently closely" and the timing of production, I
>>> assume that the signal we're looking for here is confirmation of the
>>> engineering choices.  I think that's fine (though I'm not sure this
>>> needs formal experiment status).
>>
>> I hadn't thought about this in terms of signals, but its an
>> interesting way to look at it.
>>
>> BTW, others have also said (and I agree) that lots of the things
>> suggested in the draft don't need to be a formal experiment,
>> but I think doing it that way has merit in any case.
>>
>>> But I believe we need to be really
>>> careful that it isn't mistaken for signal about the specification's
>>> quality.  It can happen that a working group has "lore" about what to
>>> do that gets folded into the implementations done by those
>>> participating, but which never quite makes it into the spec "because
>>> everyone knows it".  An implementation written during the working
>>> group process is potentially subject to this effect.
>>>
>>> An interoperating implementation written to the spec by a non-working
>>> group participant would be great signal about the specification quality,
>>> but it  is not likely to be available at the stage of the process this
>>> draft
>>> targets.
>>
>> That's very true.
>>
>> I guess I'd be a bit reluctant to try to add text to the
>> draft about this though, since it might be a bit of a rathole
>> to try discuss generic good/bad aspects of implementations
>> and specs. I could see that discussion running and running
>> and getting nowhere;-)
>>
>>
> I don't think it needs to be a rat-hole.  Let me suggest the following
> text:
> 
> OLD:
> 
>    Sometimes, it can take a long time to get a Proposed Standard
>    produced in the IETF.  This memo proposes an optional way to speed up
>    the parts of the process that happen after a WG has done its job by
>    building a "reward" for having an implementation (ideally open-
>    source) available into IETF processes.
> 
> 
> NEW:
> 
> Implementations developed during the production of an Internet-draft
> can indicate that a working group has had the opportunity to get
> early confirmation of its engineering choices.  This memo proposes
> an optional way to parallel process some final stage reviews when
> the working group management and area directors believe that the
> implementation can itself serve as a practical review of the engineering
> choices.
> 
> Does that make sense?
> 
> 
> 
>> But maybe it'd be a good idea to maintain a wiki as the
>> experiment runs where such things could be captured that
>> could feed into later evaluation of how things went.
>>
>> If that sounds useful, I'm willing to start that (maybe
>> once the draft's gotten past IETF LC) and add a bit of text
>> to the draft pointing at the wiki page. I'm also happy to
>> do some maintenance on that as things progress, if they do.
>>
>> If you think some other changes would help instead or
>> as well, I'll gladly take text of course, or suggestions
>> for how to (re-)structure the text.
>>
>>
> I think it might generally useful to talk about this in terms of
> parallel processing review, rather than issuing a reward, but
> that may be simply a a style preference.
> 
> regards,
> 
> Ted Hardie
> 
> 
>> Cheers,
>> S.
>>
>>> Again, not objecting to the experiment; I just want to be clear about
>>> what signal we believe we're getting from the implementation.
>>>
>>> Just my two cents,
>>>
>>> Ted
>>>
>>
>