Re: Guidelines for authors and reviewers

[Scott Brim <[e-mail]>]

On 5/30/08 7:54 PM, Ted Hardie allegedly wrote:
>> There are several issues that get mixed up together in defining a late
>> external review process.  By definition, this is an external review.  So
>> it is not reasonable to say that the reviewer should think like a WG
>> member, or have the full email conversations available.
> 
> No, it is not.  But it is also not reasonable to assume that each document
> carry the full context of an area with it.  There is a balance there,
> and assuming that contextualizing information should be added to
> each document on the basis of reviewer request is a problem.  In addition
> to making each document very large, they run the great risk of getting
> out of synch, with the usual hilarity resulting.

If there is _written_ context in other drafts, that's fine.  If 
_cultural_ context is missing from the drafts, that's a problem that 
should be noted and fixed.  Sometimes the drafts where the context 
should be added are already out the door, and the only place to put the 
needed context is in the draft under review.

> But we're not writing our documents for reviewers; we're writing them for
> implementors, operators, and our successors, toiling in these protocol farms,
> right?  So there is some level of common knowledge among those working
> within a protocol area we can and should assume in our documents.

You can't assume that.  We get strange questions from implementors and 
operators all the time.  Far stranger than the ones from reviewers. 
Better to assume that reviewers understand the audience as well as you 
do and what their needs will be.

> We can and should provide citations (that's what those nifty normative
> and informative references are for), but that doesn't translate into providing
> tutorial text.   I was once a big fan of it, but I see people go wrong on the basis
> of thrown-in tutorial text often enough now that I think it is often
> less than helpful and sometimes actively harmful (by causing the reader
> to believe that it is sufficient context).

Personally I agree in separating out tutorial text, but all context 
required must be supplied somewhere.

On 6/2/08 3:00 PM, Ted Hardie allegedly wrote:
> A review that is raising a showstopper has a provable or disprovable statement
> in it.  "This *will not work* in the following scenario" or even "This seems
> to have poor results in networks with high rates of non-congestive loss" creates the
> opportunity for the reviewer and working group to discuss the issue in terms
> that can be tested.   

Agreed, and reviewers with expertise do this.  Reviewers with less 
expertise may just explain their doubt and ask if it is justified, but 
even questions like that should be considered if they have not been 
already, because on occasion I've seen them save a group from disaster.

> But review comments that do not contain testable assertions end up
> being subjective.  "I think ZNK  ChillOut is better than ZNK BindMeTight,
> for the following reasons" may contain a good set of reasons, but it
> should never over-rule the consensus of a working group that has
> agreed on ZNK BindMeTight unless one of those reasons amounts to
> "it doesn't work".  

In many WGs there are multiple designs that would work, and various 
tradeoffs are made in choosing one.  There is never (afaik) a 
requirement that one approach prove the other simply does not work. 
Similarly early reviews can be very helpful even in re-examining known 
tradeoffs or bringing up new ones so that the balance between the 
approaches changes.  It is rare for a late review to change a WG 
decision in this way -- it's hard for a WG to backtrack -- but it does 
happen occasionally, and there is never a requirement that a review 
prove that a design is utterly broken, just that there are serious 
disadvantages.

If a reviewer comment is opinion based on a great deal of experience, 
for example with system design, then the reviewer -- presenting 
him/herself as having special expertise -- then has an obligation to 
recommend a solution to the problem he/she sees, in detail.

 > Comments of the type "I think Section 4 is not
> clear enough for an implementor to follow" are also subjective; they
> are very valuable and may serve as a guide to the WG/author team,
> but it is important for the reviewer to recognize that they may not
> result in change.  

Yes.

Scott
_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf