Re: Idea for a process experiment to reward running code...

[Stephen Farrell <[e-mail]>]

Hi Jari,

I agree with almost all of what you say. I think we only disagree
in two places, and perhaps more about tactics than anything else.

The first is whether or not its worthwhile addressing the specific
bit of process my draft tackles. I obviously do think it is, even
though you correctly note there's a very modest gain in time.

I think we also disagree as to whether or not an IESG statement
would have any impact - I don't think there would be any good
effect at all for any IESG statement on the topic I can envisage
that wouldn't require community consensus. And as we've seen in
the last couple of days, even my modest idea for an experiment has
generated lots of words and a hugely rough consensus if any.

That seems to be the fate of all process discussions. So my
conclusion is that the best way towards our shared goal is via
small steps such as are represented by my proposal.

I'll look at your specific comments at the end, thanks for those.

S

On 12/03/2012 10:03 PM, Jari Arkko wrote:
> Stephen,
> 
> Thanks for proposing this. You know that I agree with you that improving
> IETF's ability to publish specifications relating to real code out there
> is important. It is important to come up with ideas in this space. I
> think there has been situations where IETF has strayed from the "rough
> consensus and running code" model.
> 
> But of course following the running code is not always easy. I have some
> perspectives on this, both from the IESG side and as a document author
> who in some cases had running code. Obviously we want IETF's resources
> spent on things that result in running code, and we want experiences
> from that running code reflected back into specifications. And we want
> those fast-running implementation teams to bring their specs to the IETF
> rather than publish them somewhere else.
> 
> But on the other side, there are issues. It is sometimes difficult to
> distinguish useless junk with several implementations from useful and
> stable technology that is going to change the Internet and which also
> has several implementations. And it is difficult to guess whether an
> implementation is the beginning of something great or the by-product of
> someone's pet project that also produced the specs. In many cases it is
> very cheap to write an implementation. You can also easily build an
> implementation and fund a few other folks to build theirs. And even if
> you have independent implementations, I've seen cases where there are
> seven implementations that inter-operate but the spec is still
> unimplementable for outsiders.
> 
> If you want to support IETF work that has implementations better, you
> have to decide where your focus is in the overall process. I suspect
> that the last calls make up a very small part of IETF's overall
> end-to-end ("draft-00-to-RFC") delay. I do not have good statistics of
> this at the moment, http://www.arkko.com/tools/lifecycle/wgdocs.html has
> some numbers from buggy code that has not been run since last spring.
> Those numbers seems to indicate WG process times in the order of two
> years and LC/IESG times in the order of half a year, plus some RFC
> editor time. Intuitively, the numbers match my own experience, so maybe
> they are in the ballpark.
> 
> One of the issues is that it is very rare for document to sail through
> the LC/IESG without any changes. From the same crappy statistics,
> http://www.arkko.com/tools/admeasurements/stat/goodprocessingtime_4.html
> and http://www.arkko.com/tools/admeasurements/stat/processingtime_4.html
> that's maybe 10% of all documents; 20% in some cases but those drafts
> probably have some RFC Editor notes associated with them. A document
> that does not change goes through quite fast, maybe in a month. A
> document that needs change also needs discussion, debate, and waiting
> for a revision. Hence the half a year. Two weeks out of that? Not a big
> difference. Two weeks out of 2-3 years for the entire spec development?
> Not noticeable.
> 
> All decisions involving how you give support or priority to running code
> involve difficult judgment calls. There's running code. Does that trump
> someone's opinion that the bits should be ordered differently? That the
> design should be changed to work better through NATs? That no security
> is needed? There is no hard and fast rule for these questions, you have
> to evaluate what the situation is.
> 
> It is not easy to support running code as part of the IETF day-to-day
> process. People test obviously, on their own. Many people bring
> implementations to IETF meetings. Some organize back room meetings to do
> even proper interops. We saw some code in bits-n-bytes, and there's been
> a few other demos as well. But interops are not within IETF scope. It is
> even difficult to acquire resources for testing in all cases. In
> Atlanta, we asked for a room the hallway was no longer sufficient. We
> did this in the last minute, got some well-deserved pushback from the
> NOC crew (who did a wonderful job and supported us anyway - thanks). But
> we also got some unexpected pushback from the IESG, who among other
> things told us that they can't help us with a room in any way until the
> spec is officially a WG document. But to get to that state we'd need
> some testing to improve the spec and convince people that the stuff
> works. So catch 22. Luckily, we got a room from IPSO Alliance, who
> stepped up to help.
> 
> Anyway, all this puts me roughly in the same camp as Barry. How about
> doing this instead:
> 
> 1) Write that IESG statement or new RFC that reminds everyone that they
> should seriously consider what running code tells us when they weigh
> issues, decide to send documents forward, etc. Obviously we should not
> follow code blindly. Perhaps the statement could highlight some of the
> above issues, or the issues from Stephen's draft to make people aware of
> the complexity of the matter.
> 
> 2) Re-focus ourselves in the WG and IESG reviews on the stuff that
> actually matters. DISCUSSes on what's in the document header of the
> I-D... do not matter as much as whether the thing actually works, or is
> i18n capable or secure. Yes, I know I've erred on this myself several
> times.
> 
> 3) Remember that some of the most successful IETF specifications were
> things that were brought to us from the outside, with implementations
> and first standards revisions already done. It is not a bad model. You
> do not always have to start with a committee. You can finish with a
> committee.
> 
> 4) Find ways to support running code in all stages of the IETF process.
> Could we support unofficial interops better in our meetings, or work
> with more interop organizations? Could we use bits-n-bytes to do more
> demos, and would this be useful in promoting running code?
> 
> Some more comments on details on your draft:
> 
> * The IPR return-to-WG process is actually part of existing operations
> for all drafts. Perhaps you could clarify this.
> 
> * I'm not sure why you have taken the position that the sponsoring AD
> does not have to vote Yes in these. I think that should be necessary
> even more than with other drafts. Were you thinking that there's some
> other AD who knows the case better? There are certainly cases where
> another AD knows more about a particular case than the one officially in
> charge of the WG. In those cases it might actually make sense for that
> other AD to take on the sponsoring role. But that seems orthogonal to
> fast tracking, and is already happening.
> 
> * I think you have placed perhaps too much emphasis on verifying the
> software. I think these are judgment calls, you figure out what is
> available, what its functionality is, how it has been tested, and so on.
> It is hard to codify that in a set of rules. Strict rules may actually
> rule out some things that would be useful. For instance, would several
> proprietary implementations with well-documented interop success qualify
> or not?
> 
> Jari
> 
> 
> 
>