Re: Alternate entry document model

[Gonzalo Camarillo <[e-mail]>]

Hi,

I remember talking with Henning about the review process in the IETF
some time ago. I have pasted below the relevant section of his draft:

http://tools.ietf.org/html/draft-tschofenig-rai-reducing-delays-01#section-2

Review marathon:

      Getting a document through the IETF process does not happen
      without reviews.  There are reviews before the document becomes a
      WG item, reviews while the document is within the group, some
      chairs demand reviews before they issue a WGLC to probe
      'readiness', someone in between reviews by special expert groups
      are done (also by other SDOs and external groups, if necessary),
      then comes the WGLC, review by the responsible AD, review by the
      IESG, review by directorates (and there are many of those) and
      (depending on the type of document and history) an IETF Last Call.
      Everybody has an opinion, often not necessarily a technical
      opinion, on how documents should be written and why other solution
      approaches have not been explored.  Reviewers need time and then
      the review comments often cannot be ignored but need to be
      discussed and resolved.  When reviews happen later in the process
      then text changes are often expected to keep the reviewer happy.
      IESG members frequently put DISCUSSes on reviews and this
      increases their priority allowing a single person to, for example,
      delay the publication of a document for an extended period of
      time.  From a psychological point of view reviewers are in the
      unfortunate position that they have the feeling that something
      must be improved as an outcome of the review activity.  As soon as
      documents leave the working group the transparency is largely
      lost, despite IESG comments being sent to the authors, WG chairs
      and responsible ADs and despite information being available in the
      I-D tracker.  Mentally, many working group members consider
      documents to be 'done' when they leave the working group.

      The authors are not arguing that reviews are unnecessary but there
      has to be balance with respect to the goal that is about to be
      accomplished.


Note that this draft was written in the context of the RAI area but
seems to be equally applicable to other areas. Interestingly, the draft
also discussed the "Degeneration of the 3-level Standards Track process".

Cheers,

Gonzalo


On 31/10/2010 11:48 PM, Dave CROCKER wrote:
> 
> 
> On 10/30/2010 11:31 AM, Joel M. Halpern wrote:
>>      One of the positive effects
>> of our current system is taht because WG knows tha tthye have to clear all the
>> ADs, not just their own, they actually think about all these issues. And usually
>> manage to cope with them
> 
> A working group is diligent or it isn't.  It gets a range of feedback and 
> responds constructively to it... or it doesn't.
> 
> The working group behaviors that I have seen that pay explicit attention to the 
> specific question of satisfying AD reviews have nothing to do with quality of 
> the work and more to do with guessing what will personally bother an AD.  In 
> other words, it's about dealing with AD idiosyncrasy rather than with quality.
> 
> It is now common to get cross-area reviews and my own observation is that these 
> are a) typically quite reasonable and diligent, and b) dealt with constructive 
> by the working group.
> 
> ADs do sometimes come up with interesting and even important points, but AD 
> review is an extremely expensive and often frustrating mechanism that we already 
> have a vastly superior replacement for.  Its timing is better and it distributes 
> the work far better.
> 
> The fact that an AD sometimes catches some important problem is typically taken 
> as proof that the AD review and Discuss mechanism is essential.  This is highly 
> flawed logic, on two counts.
> 
> One is that it does not represent meaningful cost/benefit evaluation.  The cost 
> is actually quite high in energy, delay and frustration, and the significant 
> benefit overall is quite low (if the wg has been diligent and has gotten 
> cross-area reviews.)
> 
> The other is that protocol specs have a statistical likelihood of bugs, even 
> with the AD review.  We talk about AD review almost as if it ensures perfection, 
> but of course we know it does not.
> 
> Ultimately, we have to trust the real world to evaluate the safety and efficacy 
> of a protocol.  That fact ought to give us permission to balance the cost and 
> benefit of the quality assurance efforts we require during specification 
> development and approval.
> 
> 
>> Yes, it would be very good to spot all of these things sooner. I have not yet
>> seen a proposal that actually works for doing so. But letting WGs or WGs + ADs
>> approve documents for general advancement is a step likely to lead to problems.
>> If all our WGs handed their ADs high quality documents that they had checked for
>> all these issues, then maybe we could look at this differently.
> 
> We do need quality assurance efforts.  The basic idea that working group efforts 
> are subject to outside review prior to approval is a significant value-add by 
> the IETF, IMO.  The question is how to provide sufficient review in a reasonable 
> way.
> 
> I believe that cross-area reviews largely satisfy that requirement.  If 
> within-area reviews are also needed, the AD should commission them, not do them 
> directly.
> 
> d/