Re: Guidelines for authors and reviewers

[Ted Hardie <[e-mail]>]

At 4:08 PM -0700 5/30/08, Joel M. Halpern wrote:
>I have to strongly disagree with the tone of this review of the review
>document.
>(Which is not to say that Ted isn't entitled to provide the review, or
>that he is not entitled to a reasonable response.)

Tone is notoriously hard to get right, in email especially.  As an
early reviewer here, I am trying to make a forceful statement that
this is off on the wrong foot and needs to start over to make much
progress.  If that force comes off as rude, forgive me.  But this
document lacks context that you have at this point absorbed into
your bones, and I believe that without it, it will be harmful.

>
>There are several issues that get mixed up together in defining a late
>external review process.  By definition, this is an external review.  So
>it is not reasonable to say that the reviewer should think like a WG
>member, or have the full email conversations available.

No, it is not.  But it is also not reasonable to assume that each document
carry the full context of an area with it.  There is a balance there,
and assuming that contextualizing information should be added to
each document on the basis of reviewer request is a problem.  In addition
to making each document very large, they run the great risk of getting
out of synch, with the usual hilarity resulting.

>So, the first question is what should happen when a reviewer says "X
>does not make sense" or "Y seems to be assumed without explanation."
>Some of the time, that is because that context is provided by other
>documents which are reasonable to have read.  Someone needs to say that.
>  (If a reviewer has become confused, likely some AD sill also become
>confused.  Maybe the sponsoring AD will realize that the solution is
>some other document that is already referenced.  But then again, maybe
>he won't.)

>An awful lot of the time though, the issue is one of "common knowledge".
>  Sometimes bluntly stated as "all the participants understand that Z,
>which means ..."  Well, the fact that working group members know Z does
>not mean taht readers know it if it is not part of some other document
>that is a reasonable context.  (The OSPF base document is reasonable
>assumed context for an OSPF extension.  But the full set of all
>published BGP extensions that are not mentioned in the document is not
>suitable context for a BGP extensions.  Fortunately, the recent
>extension I reviewed did state what document was needed for context.)
>If the reviewer is confused, it is reasoanble to assume that other
>readers will be confused.

But we're not writing our documents for reviewers; we're writing them for
implementors, operators, and our successors, toiling in these protocol farms,
right?  So there is some level of common knowledge among those working
within a protocol area we can and should assume in our documents.
We can and should provide citations (that's what those nifty normative
and informative references are for), but that doesn't translate into providing
tutorial text.   I was once a big fan of it, but I see people go wrong on the basis
of thrown-in tutorial text often enough now that I think it is often
less than helpful and sometimes actively harmful (by causing the reader
to believe that it is sufficient context).



>On design decisions, there is an even more complex tradeoff.  I have
>reviewed several documents which had questionable design decisions.  In
>one review I recently wrote that I did not expect the following comment
>to change the WG consensus, but that I considered the specific mechanism
>used by the document a bad idea.  If I had not known that the particular
>mechanism had been discussed, I might have put it more forcefully.
>On the other hand, a while back I reviewed a document which had a
>mechanism which, although the working group had indeed agreed on it,
>fundamentally didn't work.  I don't care how much they agreed.  It
>needed to be changed.  And they changed it.  (It was incumbent upon me
>to provide a clear and coherent explanation of why it was broken.)

These both sound like excellent reviews:  you expressed your personal
design preferences in the first instance but did not try to force it over
the consensus of the working group, and pointed out a showstopper
in the second. 

Now, show me in this draft how these two cases are distinguished,
which is critical to getting a review done right? 


>The problem I see is that working groups make all sorts of interesting
>mistakes.  Sometimes deliberately, sometimes by accident.  The job of
>the external reviewer is to look for such things.

Mistakes are not "does not agree with my prejudices", and that is a key
problem in many of the reviews we actually see. 



>Given that we have a policy that all comments deserve responses, it
>seems particularly appropriate that review team comments better get
>responses.

Uh, why?  If you up and decide to read every 17th Last Call for your
own edification and respond with comments  that are reasonable reviews,
why should someone else's comments get better responses than yours?


>
>Also, on the time frames, it is worth noting that these are late
>reviews.  If you are lucky, they are produced early in the LC cycle.
>But even reviewers are human and take some time.  But if the review is
>not responded to in a timely fashion, then other folks are likely to
>assume that there is a real problem that needs to be addressed, rather
>than finding out that the reviewer simply misread the document
>(reviewers make mistakes too.)
>
>The whole tone of your comment seems to be "these should have been made
>during and as part of the WG process.  But late external reviews benefit
>from being external.  (When I write a paper, I like outside review
>because I get used to my own writing and assume that things flow.
>Sometimes they don't   Working groups make the same mistakes sometimes.)

My comment is not simply that late reviews should be made earlier.  It is that
we need to distinguish between issues that a reviewer can and should raise
as showstoppers "Will not work| Will break Internet | Uses component parts in
ways will cause whole to fall apart" and those that question the decisions
of the working group when none of those conditions are fulfilled.  That kind
of issue has a time and a place; document development (and, if necessary, the
bis or ter versions) is where that should happen.  If you push for ZNK
ChillOut over BindMeTight in a late review, in other words, you need to show
that BindMeTight doesn't work.  If you push it over the consensus of the working
group at late stage, you should be satisfied with a "considered; choice made"
and nothing else; anything else puts the power for determining the shape of
the document in the hands of those doing the review instead of doing the work.
That's wrong, that's dangerous, and any document that seems to push that
point of view needs to be extensively revised or dropped.

To help with the tone issue, permit me to wish you a happy weekend,
				regards,
					Ted


>Yours,
>Joel M. Halpern
>
>Ted Hardie wrote:
>> At 3:04 PM -0700 5/29/08, Suresh Krishnan wrote:
>>> Hi Folks,
>>>   We have written a draft describing some guidelines for authors and
>>> reviewers of internet drafts. We would really appreciate it if you can
>>> spend some time to go over it and provide comments and/or suggestions
>>> for improvement.
>>>
>>> Thanks
>>> Suresh, Pasi and Eric
>>
>> Hi Suresh, Pasi, Eric,
>>
>>       I looked at it, and, while I laud any efforts to get folks to review
>> things effectively, I have to say that I found this to be a pretty drafty draft.
>> It does not reference the Tao, 2026, or any of the developed educational materials;
>> its only listed reference, in fact, is 2119 and that does not seem to be referenced
>> within the text.  That means that this comes across as pretty context free.
>> It needs anchoring to the rest of our processes.
>>       One of the things that I believe that anchoring should provide
>> is a pretty significant change of perspective.  As this reads now,
> > it implies a lot of power in the hands of reviews to elicit (or even require)
>> change.   It seems to want document authors to accede to requests for
>> tutorial material as a matter of course and to significant technical changes
>> with a modicum of fuss.  That's not the right approach.  The outcome of our
>> document development should not be a negotiation between the authors
>> and the assigned reviewers.  It should be a conversation in the working group
>> among those who will actually develop the implementations, those who
>> will deploy it, and those who are affected by the system of which the
>> documented technology  is a part.
>>       Reviews that work to relate a particular document's technology
>> to the larger whole of which it is a part (asking: how does this impact
>> congestion control in the access network or core, use deployed security systems,
>> relate to the identifier mechanisms common to URIs, etc.) are very valuable.
>> But many reviews represent questions about decisions that come down to
>> design choices that working groups should have the power to make without
>> extensive second-guessing.  Folks who want to have a say at the level can and
>> should do so with the simple method of joining the working group list and commenting
>> as part of the general development.  That's not "review" (of someone else's work)
>> it is "participation" (in joint work), and it is fundamentally more valuable.
>> Without the context of how "participation" works, your documents description
>> of "review" comes off very badly indeed.  I hope that future versions can correct
>> that and place review within a broader, more productive context.
>>                       good luck,
>>                               Ted
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> -------- Original Message --------
>>> Subject: I-D Action:draft-krishnan-review-process-00.txt
>>> Date: Wed, 28 May 2008 11:15:01 -0700 (PDT)
>>> From: [e-mail]
>>> Reply-To: [e-mail]
>>> To: [e-mail]
>>>
>>> A New Internet-Draft is available from the on-line Internet-Drafts
>>> directories.
>>>
>>>        Title           : Guidelines to authors and reviewers regarding the
>>> IETF review process
>>>        Author(s)       : S. Krishnan, et al.
>>>        Filename        : draft-krishnan-review-process-00.txt
>>>        Pages           : 10
>>>        Date            : 2008-05-28
>>>
>>> This document describes the IETF review process and provides
>>> guidelines to draft authors and reviewers on how to effectively
>>> participate in it.
>>>
>>> A URL for this Internet-Draft is:
>>> http://www.ietf.org/internet-drafts/draft-krishnan-review-process-00.txt
>>>
>>> Internet-Drafts are also available by anonymous FTP at:
>>> ftp://ftp.ietf.org/internet-drafts/
>>>
>>> Below is the data which will enable a MIME compliant mail reader
>>> implementation to automatically retrieve the ASCII version of the
>>> Internet-Draft.
>>>
>>> _______________________________________________
>>> IETF mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/ietf
>>
>> _______________________________________________
>> IETF mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ietf
>>

_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf