Re: Guidelines for authors and reviewers

[Suresh Krishnan <[e-mail]>]

Hi Ted,
   Please find responses inline

Ted Hardie wrote:
> At 2:42 PM -0700 5/30/08, Suresh Krishnan wrote:
>> I do share your sentiment, but this is not what we set out to do
>> originally. We set out to document the review process(es) and provide
>> some guidelines to effectively use the received reviews. I think the
>> right document to weave the stuff together would be the Tao. But I do
>> agree that adding references to the Tao and 2026 make sense. The
>> dangling reference to 2119 is an editorial oversight :-).
> 
> If the Tao is the right document to do the weaving in, then the right thing
> to do may well be incorporate more discussion of the review
> process in the next version of the Tao.  A stand-alone document which
> says "to get this context go read this first" might work, but it runs
> the usual risk.  And statements like:
> 
> "reviews also play an important part in how IETF determines
>    consensus. "
> 
> really need to be related to our process documents.  Personally, I
> don't think that's actually either true or the point.  Many solicited
> reviews  are meant to discover technical show stoppers or areas where
> the impact of technology on other parts of 'net have not been
> taken into account.  Determination of consensus is different,
> and it has a large amount to do with our participation model.
> Changes to the pool of folks from whom consensus is
> solicited should not be done lightly in informational documents.

I think there is some context missing here. This is what we talk about 
in section 4.1.

> 
> 
>>>       One of the things that I believe that anchoring should provide
>>> is a pretty significant change of perspective.  As this reads now,
>>> it implies a lot of power in the hands of reviews to elicit (or even require)
>>> change.   It seems to want document authors to accede to requests for
>>> tutorial material as a matter of course and to significant technical changes
>> I am sorry you feel that way. It was not the intent of the document. The
>> document just states that a concern raised in the review deserves a
>> response (not necessarily a change). One valid response is "Mr.
>> reviewer, what you raised is not a problem. This was a design decision
>> taken by the WG and the discussion thread about this is at
>> http://mlarchive.invalid/msgfoo.html".
> 
> The tutorial material comment comes from this section:
> 
>       If the reviewer has not understood some part of the draft, a very
>       common response is to explain the topic in email.  However, often
>       that explanation should really be in the draft itself, not just in
>       emails to the authors, so that future readers will also understand
>       the text.

This is from my personal experience. I have been asked by reviewers 
(including IESG members) to add such text in the past. For the sake of 
clarity, I complied. I do not see the downside of this.

> 
> The key missing piece here is that the effort to get outside reviewers
> means that they are missing context that those actually using the
> document will likely have.  Asking that it be included as a matter of
> course on the query of a reviewer produces changes that then have to
> be reviewed and agreed to by the working group.  That's a lot of work
> for a lot of people for things which may well be obvious to those
> within the group using a technology. 

If someone reads a document and the normative references, and does not 
understand something, it needs to be fixed. RFCs are not only for people 
who worked on creating the technology.

> "
> The document's description of politeness in responses also sets
> timeframes that sound a lot like deadlines:
> 
> "two or three business days might be a
>    reasonable amount of time to take for a response.  It is possible
>    that the authors may be unavailable to respond to a review within
>    this timeline since they may be sick/on vacation/busy with dayjob
>    etc.  In this case, the document shepherd (if any) can respond to the
>    review and follow up with the authors at a later time."
> 
> The ability to force an interrupt on busy people (either authors or shepherds)
> is real power, and we need to be careful how we cast that.  A comment on a document
> (and that is what reviews are) does deserve to be considered, but the extent
> to which a response is required and the timeframe for that response is much
> more fluid than this implies.  Bundling all comments received during a
> Last Call, considering them as an author team, then issuing a new draft
> or set of RFC Editor notes (or, honestly, punting them all) may be the most
> effective thing to do.  In that case, the only early response you'll
> get is "message received".  If you want that, permit me to suggest
> the handy MDN functionality likely available in your mail client.

I can remove this from the document. This is the amount of time that I 
take on average to respond to a review. But, let's say somebody writes a 
draft for the first time ever in her life. She gets a review from you on 
her draft. She currently has no idea how much time she has to respond to 
you. The options being

* Reasonable amount of time (specified in an IETF process document: not 
this one)
* Reasonable amount of time (as deemed appropriate by the authors)

Since we limit the amount of time people have to send in the reviews we 
need to limit the time people take to respond to reviews. Otherwise, 
reviews just fall through the cracks and end up being useless.

> 
> 
>>> with a modicum of fuss.  That's not the right approach.  The outcome of our
>>> document development should not be a negotiation between the authors
>>> and the assigned reviewers.  It should be a conversation in the working group
>>> among those who will actually develop the implementations, those who
>>> will deploy it, and those who are affected by the system of which the
>>> documented technology  is a part.
>> Agree. And hence this text in section 3.5
>>
>> "   After the author and reviewer agree that the issues have been fixed,
>>    for working group documents, substantial issues need to be confirmed
>>    on the mailing list.  Once the document has entered IESG processing,
>>    new versions should not be posted unless the document shepherd or AD
>>    explicitly says so."
> 
> You say "agree", but the text you point to says something substantially
> different from what I did.  The text you point to says, in essence,
> "after the reviewer and document author have finished their negotiation,
> the mailing list should confirm".  That's exactly wrong.

I do see the difference, but I quoted the text to show that I agree with 
your text that "The outcome of our document development should not be a 
negotiation between the authors and the assigned reviewers". The WG has 
the final say. When the WG gets involved is where we disagree.

> 
> 
>>>       Reviews that work to relate a particular document's technology
>>> to the larger whole of which it is a part (asking: how does this impact
>>> congestion control in the access network or core, use deployed security systems,
>>> relate to the identifier mechanisms common to URIs, etc.) are very valuable.
>>> But many reviews represent questions about decisions that come down to
>>> design choices that working groups should have the power to make without
>>> extensive second-guessing.  Folks who want to have a say at the level can and
>>> should do so with the simple method of joining the working group list and commenting
>>> as part of the general development.  That's not "review" (of someone else's work)
>>> it is "participation" (in joint work), and it is fundamentally more valuable.
>> Agree in principle. But a "late" reviewer usually gets documents from
>> multiple wgs and subscribing to the MLs just to raise one issue can get
>> to be a tiring endeavor. Last year, I reviewed documents from 20
>> different WGs (apart from the ones I am usually involved in). I cannot
>> handle mailing list traffic from 20 more mailing lists.
> 
> I must not have been clear.  I am not saying that a reviewer must subscribe
> to a mailing list to raise an issue.  I am saying there are two classes of
> comments: 
> 
> 1) issues raised about the impact of the technology on the larger 'net
> 	or the usage of the technology within a larger context;
> 
> 2) discussion of design decisions. 
> 
> Late comments of type 2 by folks who did not participate in the design
> discussions should expect fairly short replies; the right time and place
> to involve yourself in that is during the participation phase.  If you
> deeply care that folks using ZNK use its ChillOut encoding when describing gttp
> extensions, you can  join the gttp mailing list and argue the point.  If you
> come as a reviewer and find they have chosen  ZNK using its older
> BindMeTight encoding, you may ask something like "Were the trade-offs
> between BindMeTight and ChillOut considered?" and be satisfied
> with "Yep" (or prepared to help if they say "What's ChillOut?").  But if
> you push back on the use of BindMeTight, you had better be able to
> say how using it causes the protocol to fail to meet its goals.  Anything
> less is not an appropriate late-stage review comment.  It comes
> across as an attempt to subvert the process for attaining consensus.

I think a late stage reviewer should be able to ask for the rationale of 
choosing BindMeTight over ChillOut. The rationale should be available 
either in the document itself or on a publicly accessible archive. 
Section 4.1 provides guidelines for reviewers about the level of review. 
  This is the text about "late" reviews

"The reviews that occur after the
    draft has been progressed from the working group have a very
    different perspective on the contents of the draft.  They explore
    very different aspects of the draft such as security aspects,
    operational aspects, readability etc. that are not usually on the top
    of the list of priorities of the authors and the early reviewers.
    They also explore how the mechanisms proposed in the document fit
    into the Internet architecture as a whole and what detrimental
    effects they will have, if any."

What do you disagree with here?

> 
>> If this were
>> mandated, I would rather not review the document. I would also like to
>> point out that most of the reviews received during IETF Last Call are of
>> the nature you describe.
> 
> I tried to describe two different types, so it is not clear to me which you
> mean here.

I meant 2).

> 
>>> Without the context of how "participation" works, your documents description
>>> of "review" comes off very badly indeed.  I hope that future versions can correct
>>> that and place review within a broader, more productive context.
>> I personally feel that the TAO (4677 and descendants) should be the
>> document that sets the various pieces in context, but I am open to
>> suggestions on how to go about fitting this document into a broader
>> context.
> 
> I have obviously not been clear here either, so permit this bluntness:
> as it stands now,this document is harmful.  It fails to place the process
> in sufficient context.  The result is a description that over-emphasizes
> late stage review and satisfying those who conduct those reviews.
> It needs to be re-written using a full context, folded into a document
> that establishes that context, or abandoned.  Having it go forward
> alone, without that context, is not appropriate.

I am open to suggestions on how to go forward with this document. This 
is just version -00 of the document. It is pretty much table of contents 
and preliminary thoughts. Give it some time to mature.

> 
> To end on a high note, it is beautifully formatted.

:-)

Thanks
Suresh
_______________________________________________
IETF mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ietf