Re: Guidelines for authors and reviewers

["Joel M. Halpern" <[e-mail]>]

I have to strongly disagree with the tone of this review of the review 
document.
(Which is not to say that Ted isn't entitled to provide the review, or 
that he is not entitled to a reasonable response.)

There are several issues that get mixed up together in defining a late 
external review process.  By definition, this is an external review.  So 
it is not reasonable to say that the reviewer should think like a WG 
member, or have the full email conversations available.

So, the first question is what should happen when a reviewer says "X 
does not make sense" or "Y seems to be assumed without explanation." 
Some of the time, that is because that context is provided by other 
documents which are reasonable to have read.  Someone needs to say that. 
  (If a reviewer has become confused, likely some AD sill also become 
confused.  Maybe the sponsoring AD will realize that the solution is 
some other document that is already referenced.  But then again, maybe 
he won't.)
An awful lot of the time though, the issue is one of "common knowledge". 
  Sometimes bluntly stated as "all the participants understand that Z, 
which means ..."  Well, the fact that working group members know Z does 
not mean taht readers know it if it is not part of some other document 
that is a reasonable context.  (The OSPF base document is reasonable 
assumed context for an OSPF extension.  But the full set of all 
published BGP extensions that are not mentioned in the document is not 
suitable context for a BGP extensions.  Fortunately, the recent 
extension I reviewed did state what document was needed for context.)
If the reviewer is confused, it is reasoanble to assume that other 
readers will be confused.

On design decisions, there is an evenr more complex tradeoff.  I have 
reviewed several documents which had questionable design decisions.  In 
one review I recently wrote that I did not expect the following comment 
to change the WG consensus, but that I considered the specific mechanism 
used by the document a bad idea.  If I had not known that the particular 
mechanism had been discussed, I might have put it more forcefully.
On the other hand, a while back I reviewed a document which had a 
mechanism which, although the working group had indeed agreed on it, 
fundamentally didn't work.  I don't care how much they agreed.  It 
needed to be changed.  And they changed it.  (It was incumbent upon me 
to provide a clear and coherent explanation of why it was broken.)

The problem I see is that working groups make all sorts of interesting 
mistakes.  Sometimes deliberately, sometimes by accident.  The job of 
the external reviewer is to look for such things.
Given that we have a policy that all comments deserve responses, it 
seems particularly appropriate that review team comments better get 
responses.

Also, on the time frames, it is worth noting that these are late 
reviews.  If you are lucky, they are produced early in the LC cycle. 
But even reviewers are human and take some time.  But if the review is 
not responded to in a timely fashion, then other folks are likely to 
assume that there is a real problem that needs to be addressed, rather 
than finding out that the reviewer simply misread the document 
(reviewers make mistakes too.)

The whole tone of your comment seems to be "these should have been made 
during and as part of the WG process.  But late external reviews benefit 
from being external.  (When I write a paper, I like outside review 
because I get used to my own writing and assume that things flow. 
Sometimes they don't   Working groups make the same mistakes sometimes.)

Yours,
Joel M. Halpern

Ted Hardie wrote:
> At 3:04 PM -0700 5/29/08, Suresh Krishnan wrote:
>> Hi Folks,
>>   We have written a draft describing some guidelines for authors and
>> reviewers of internet drafts. We would really appreciate it if you can
>> spend some time to go over it and provide comments and/or suggestions
>> for improvement.
>>
>> Thanks
>> Suresh, Pasi and Eric
> 
> Hi Suresh, Pasi, Eric,
> 
> 	I looked at it, and, while I laud any efforts to get folks to review
> things effectively, I have to say that I found this to be a pretty drafty draft.
> It does not reference the Tao, 2026, or any of the developed educational materials;
> its only listed reference, in fact, is 2119 and that does not seem to be referenced
> within the text.  That means that this comes across as pretty context free.
> It needs anchoring to the rest of our processes.
> 	One of the things that I believe that anchoring should provide
> is a pretty significant change of perspective.  As this reads now,
> it implies a lot of power in the hands of reviews to elicit (or even require)
> change.   It seems to want document authors to accede to requests for
> tutorial material as a matter of course and to significant technical changes
> with a modicum of fuss.  That's not the right approach.  The outcome of our
> document development should not be a negotiation between the authors
> and the assigned reviewers.  It should be a conversation in the working group
> among those who will actually develop the implementations, those who
> will deploy it, and those who are affected by the system of which the
> documented technology  is a part. 
> 	Reviews that work to relate a particular document's technology
> to the larger whole of which it is a part (asking: how does this impact
> congestion control in the access network or core, use deployed security systems,
> relate to the identifier mechanisms common to URIs, etc.) are very valuable.
> But many reviews represent questions about decisions that come down to
> design choices that working groups should have the power to make without
> extensive second-guessing.  Folks who want to have a say at the level can and
> should do so with the simple method of joining the working group list and commenting
> as part of the general development.  That's not "review" (of someone else's work)
> it is "participation" (in joint work), and it is fundamentally more valuable.
> Without the context of how "participation" works, your documents description
> of "review" comes off very badly indeed.  I hope that future versions can correct
> that and place review within a broader, more productive context.
> 			good luck,
> 				Ted
> 
>  
> 
> 
> 
> 
> 
> 
> 
> 
>> -------- Original Message --------
>> Subject: I-D Action:draft-krishnan-review-process-00.txt
>> Date: Wed, 28 May 2008 11:15:01 -0700 (PDT)
>> From: [e-mail]
>> Reply-To: [e-mail]
>> To: [e-mail]
>>
>> A New Internet-Draft is available from the on-line Internet-Drafts
>> directories.
>>
>>        Title           : Guidelines to authors and reviewers regarding the
>> IETF review process
>>        Author(s)       : S. Krishnan, et al.
>>        Filename        : draft-krishnan-review-process-00.txt
>>        Pages           : 10
>>        Date            : 2008-05-28
>>
>> This document describes the IETF review process and provides
>> guidelines to draft authors and reviewers on how to effectively
>> participate in it.
>>
>> A URL for this Internet-Draft is:
>> http://www.ietf.org/internet-drafts/draft-krishnan-review-process-00.txt
>>
>> Internet-Drafts are also available by anonymous FTP at:
>> ftp://ftp.ietf.org/internet-drafts/
>>
>> Below is the data which will enable a MIME compliant mail reader
>> implementation to automatically retrieve the ASCII version of the
>> Internet-Draft.
>>
>> _______________________________________________
>> IETF mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ietf
> 
> _______________________________________________
> IETF mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ietf
> 
_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf