Re: Guidelines for authors and reviewers

[Ted Hardie <[e-mail]>]

At 2:42 PM -0700 5/30/08, Suresh Krishnan wrote:
>
>I do share your sentiment, but this is not what we set out to do
>originally. We set out to document the review process(es) and provide
>some guidelines to effectively use the received reviews. I think the
>right document to weave the stuff together would be the Tao. But I do
>agree that adding references to the Tao and 2026 make sense. The
>dangling reference to 2119 is an editorial oversight :-).

If the Tao is the right document to do the weaving in, then the right thing
to do may well be incorporate more discussion of the review
process in the next version of the Tao.  A stand-alone document which
says "to get this context go read this first" might work, but it runs
the usual risk.  And statements like:

"reviews also play an important part in how IETF determines
   consensus. "

really need to be related to our process documents.  Personally, I
don't think that's actually either true or the point.  Many solicited
reviews  are meant to discover technical show stoppers or areas where
the impact of technology on other parts of 'net have not been
taken into account.  Determination of consensus is different,
and it has a large amount to do with our participation model.
Changes to the pool of folks from whom consensus is
solicited should not be done lightly in informational documents.


> >       One of the things that I believe that anchoring should provide
>> is a pretty significant change of perspective.  As this reads now,
>> it implies a lot of power in the hands of reviews to elicit (or even require)
>> change.   It seems to want document authors to accede to requests for
>> tutorial material as a matter of course and to significant technical changes
>
>I am sorry you feel that way. It was not the intent of the document. The
>document just states that a concern raised in the review deserves a
>response (not necessarily a change). One valid response is "Mr.
>reviewer, what you raised is not a problem. This was a design decision
>taken by the WG and the discussion thread about this is at
>http://mlarchive.invalid/msgfoo.html".

The tutorial material comment comes from this section:

      If the reviewer has not understood some part of the draft, a very
      common response is to explain the topic in email.  However, often
      that explanation should really be in the draft itself, not just in
      emails to the authors, so that future readers will also understand
      the text.

The key missing piece here is that the effort to get outside reviewers
means that they are missing context that those actually using the
document will likely have.  Asking that it be included as a matter of
course on the query of a reviewer produces changes that then have to
be reviewed and agreed to by the working group.  That's a lot of work
for a lot of people for things which may well be obvious to those
within the group using a technology. 
"
The document's description of politeness in responses also sets
timeframes that sound a lot like deadlines:

"two or three business days might be a
   reasonable amount of time to take for a response.  It is possible
   that the authors may be unavailable to respond to a review within
   this timeline since they may be sick/on vacation/busy with dayjob
   etc.  In this case, the document shepherd (if any) can respond to the
   review and follow up with the authors at a later time."

The ability to force an interrupt on busy people (either authors or shepherds)
is real power, and we need to be careful how we cast that.  A comment on a document
(and that is what reviews are) does deserve to be considered, but the extent
to which a response is required and the timeframe for that response is much
more fluid than this implies.  Bundling all comments received during a
Last Call, considering them as an author team, then issuing a new draft
or set of RFC Editor notes (or, honestly, punting them all) may be the most
effective thing to do.  In that case, the only early response you'll
get is "message received".  If you want that, permit me to suggest
the handy MDN functionality likely available in your mail client.


> > with a modicum of fuss.  That's not the right approach.  The outcome of our
>> document development should not be a negotiation between the authors
>> and the assigned reviewers.  It should be a conversation in the working group
>> among those who will actually develop the implementations, those who
>> will deploy it, and those who are affected by the system of which the
>> documented technology  is a part.
>
>Agree. And hence this text in section 3.5
>
>"   After the author and reviewer agree that the issues have been fixed,
>    for working group documents, substantial issues need to be confirmed
>    on the mailing list.  Once the document has entered IESG processing,
>    new versions should not be posted unless the document shepherd or AD
>    explicitly says so."

You say "agree", but the text you point to says something substantially
different from what I did.  The text you point to says, in essence,
"after the reviewer and document author have finished their negotiation,
the mailing list should confirm".  That's exactly wrong.


> >       Reviews that work to relate a particular document's technology
>> to the larger whole of which it is a part (asking: how does this impact
>> congestion control in the access network or core, use deployed security systems,
>> relate to the identifier mechanisms common to URIs, etc.) are very valuable.
>> But many reviews represent questions about decisions that come down to
>> design choices that working groups should have the power to make without
>> extensive second-guessing.  Folks who want to have a say at the level can and
>> should do so with the simple method of joining the working group list and commenting
>> as part of the general development.  That's not "review" (of someone else's work)
>> it is "participation" (in joint work), and it is fundamentally more valuable.
>
>Agree in principle. But a "late" reviewer usually gets documents from
>multiple wgs and subscribing to the MLs just to raise one issue can get
>to be a tiring endeavor. Last year, I reviewed documents from 20
>different WGs (apart from the ones I am usually involved in). I cannot
>handle mailing list traffic from 20 more mailing lists.

I must not have been clear.  I am not saying that a reviewer must subscribe
to a mailing list to raise an issue.  I am saying there are two classes of
comments: 

1) issues raised about the impact of the technology on the larger 'net
	or the usage of the technology within a larger context;

2) discussion of design decisions. 

Late comments of type 2 by folks who did not participate in the design
discussions should expect fairly short replies; the right time and place
to involve yourself in that is during the participation phase.  If you
deeply care that folks using ZNK use its ChillOut encoding when describing gttp
extensions, you can  join the gttp mailing list and argue the point.  If you
come as a reviewer and find they have chosen  ZNK using its older
BindMeTight encoding, you may ask something like "Were the trade-offs
between BindMeTight and ChillOut considered?" and be satisfied
with "Yep" (or prepared to help if they say "What's ChillOut?").  But if
you push back on the use of BindMeTight, you had better be able to
say how using it causes the protocol to fail to meet its goals.  Anything
less is not an appropriate late-stage review comment.  It comes
across as an attempt to subvert the process for attaining consensus.

>If this were
>mandated, I would rather not review the document. I would also like to
>point out that most of the reviews received during IETF Last Call are of
>the nature you describe.

I tried to describe two different types, so it is not clear to me which you
mean here.

> > Without the context of how "participation" works, your documents description
>> of "review" comes off very badly indeed.  I hope that future versions can correct
>> that and place review within a broader, more productive context.
>
>I personally feel that the TAO (4677 and descendants) should be the
>document that sets the various pieces in context, but I am open to
>suggestions on how to go about fitting this document into a broader
>context.

I have obviously not been clear here either, so permit this bluntness:
as it stands now,this document is harmful.  It fails to place the process
in sufficient context.  The result is a description that over-emphasizes
late stage review and satisfying those who conduct those reviews.
It needs to be re-written using a full context, folded into a document
that establishes that context, or abandoned.  Having it go forward
alone, without that context, is not appropriate.

To end on a high note, it is beautifully formatted.

				regards,
					Ted Hardie






>Thanks
>Suresh

_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf