Re: Guidelines for authors and reviewers

"Joel M. Halpern" <jmh@joelhalpern.com> Mon, 02 June 2008 19:38 UTC

Return-Path: <ietf-bounces@ietf.org>
X-Original-To: ietf-archive@megatron.ietf.org
Delivered-To: ietfarch-ietf-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6B16F3A69A9; Mon, 2 Jun 2008 12:38:17 -0700 (PDT)
X-Original-To: ietf@core3.amsl.com
Delivered-To: ietf@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5538328C160 for <ietf@core3.amsl.com>; Mon, 2 Jun 2008 12:38:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kUWPapJKVRnm for <ietf@core3.amsl.com>; Mon, 2 Jun 2008 12:38:15 -0700 (PDT)
Received: from bender-mail.tigertech.net (bender-mail.tigertech.net [64.62.209.30]) by core3.amsl.com (Postfix) with ESMTP id 01A033A69A9 for <ietf@ietf.org>; Mon, 2 Jun 2008 12:38:14 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by bender.tigertech.net (Postfix) with ESMTP id CF3087DED; Mon, 2 Jun 2008 12:38:16 -0700 (PDT)
X-Virus-Scanned: Debian amavisd-new at bender.tigertech.net
Received: from [192.168.1.3] (pool-71-163-24-2.washdc.fios.verizon.net [71.163.24.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bender.tigertech.net (Postfix) with ESMTP id 7724D7DE9; Mon, 2 Jun 2008 12:38:14 -0700 (PDT)
Message-ID: <48444C08.3010203@joelhalpern.com>
Date: Mon, 02 Jun 2008 15:37:44 -0400
From: "Joel M. Halpern" <jmh@joelhalpern.com>
User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)
MIME-Version: 1.0
To: Ted Hardie <hardie@qualcomm.com>
Subject: Re: Guidelines for authors and reviewers
References: <483F2881.40306@ericsson.com> <p06240601c465eaec8585@[129.46.226.27]> <484088F5.8080808@joelhalpern.com> <p0624060ac4663bc97930@[129.46.226.27]> <484097E7.3080603@joelhalpern.com> <p06240602c469eabf7c90@[129.46.226.27]>
In-Reply-To: <p06240602c469eabf7c90@[129.46.226.27]>
Cc: IETF Discussion <ietf@ietf.org>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: IETF Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/ietf>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: ietf-bounces@ietf.org
Errors-To: ietf-bounces@ietf.org

Trying to paraphrase Ted's response to my comments, and then reply. 
(The paraphrase is to try to make sure that I understood the comment.)

I think that Ted is saying that review comments should have testable, 
actionable comments.  Further, if I am reading him right, he is saying 
that comments that are not testable are probably opinion.

That does seem a good place to start.  Further, Ted comments taht 
reviewers should understand that not all their comments are going to be 
accepted.  I agree.  Not even all of their comments based on proper 
understanding of the document are going to be accepted.

But, the question of where the boundary is between "that does not work" 
and "in my opinion that is a bad idea"  is, in my experience, frequently 
fuzzy.  To use a concrete example, we have plenty of experience to show 
that using a field length to determine what kind of field something has 
is a bad idea.  It fails, over time, in a myriad ways.  But no matter 
how many horror stories I put together, I was going to lose that 
particular debate with that particular document.
On the other hand, there are things which look perfect in theory, but we 
know turn out to be more fragile and are therefore bad ideas.  The 
assertion that such a thing is a bad idea is going to look like opinion, 
but is useful.

In fact, part of my concern is that sometimes opinions can be as 
important as facts.  Yes, reviewers need to be prepared to accept 
push-back.  But it should not be against the rules for a reviewer to say 
"I have seen exactly this before, and by the time they made it work it 
was a mess.  Please don't do this."

And even if it is opinion, it deserves a response.

TO use another example, when I said to a working group "Having worked 
with a lot of testers, I think that they will have trouble following the 
instructions as you have written them." I was clearly stating an 
opinion.  But it was an opinion that at least deserved a response.  (In 
fact, the working group did respond, and a good compromise was reached 
on that and a number of related issues.)

Part of this is that just as we expect authors to respond politely to 
reviewers, reviewers have an obligation of politeness towards the WG. 
Reviewers should assume that odd things are there for good reasons, even 
while raising the question as to whether they belong.

But trying to create a hard line between opinion and fact is I believe 
counter-productive.  If the issue is larger than it looks on reading the 
review, then the reviewer can add more explanation, examples, or 
justification in the dialog with the authors / WG.  Similarly, if the WG 
is overreacting to something but feels it is a big deal, the reviewer 
can say in followup "I had not realized that would be a big deal.  Let 
it be."

We are all presumably adults.  We are also human.


There is also a subtext that comes up periodically about the comments 
being actionable.  This has been raised for outside reviews and IESG 
reviews.  While I certainly agree that actionable comments are much 
better, it is not always possible.  If the problem is that the reader 
can not figure out what the text means, it is hard to suggest a fix.  It 
also gets laborious if the text could mean one of three different 
things.  Should the reviewer have to provide text for all three 
alternative meanings?  After all "clarify the text" while clear is not 
very specific as to what action is needed.  There are also other cases 
that come up where it may be impractical to require actionable comments.

Yours,
Joel M. Halpern

Ted Hardie wrote:
> At 5:12 PM -0700 5/30/08, Joel M. Halpern wrote:
>>> These both sound like excellent reviews:  you expressed your personal
>>> design preferences in the first instance but did not try to force it over
>>> the consensus of the working group, and pointed out a showstopper
>>> in the second.
>>>
>>> Now, show me in this draft how these two cases are distinguished,
>>> which is critical to getting a review done right?
>> The problem I have is that I do not know how to write text in a draft
>> that distinguishes those two.  The line between them is very tricky, and
>> possibly subjective.
> 
> Here is where I think we may have some room for some fruitful discussion.
> 
> A review that is raising a showstopper has a provable or disprovable statement
> in it.  "This *will not work* in the following scenario" or even "This seems
> to have poor results in networks with high rates of non-congestive loss" creates the
> opportunity for the reviewer and working group to discuss the issue in terms
> that can be tested.   The working group may argue (and, presumably, demonstrate)
> that either statement is false, but there is a way to move forward that is not
> just iteration of position.  To put that task to the working group, a reviewer
> should be able to back their statement with a reasonably clear, worked set of
> reasons for the assertion.   A working group can disprove the statement,
> indicate why an applicability statement is a better response ( demonstrating
> that the cases where it does work or does have reasonable results are useful
> enough and distinguishable enough that the document should go forward),
> or make relevant technical changes.
> 
> But review comments that do not contain testable assertions end up
> being subjective.  "I think ZNK  ChillOut is better than ZNK BindMeTight,
> for the following reasons" may contain a good set of reasons, but it
> should never over-rule the consensus of a working group that has
> agreed on ZNK BindMeTight unless one of those reasons amounts to
> "it doesn't work".  Comments of the type "I think Section 4 is not
> clear enough for an implementor to follow" are also subjective; they
> are very valuable and may serve as a guide to the WG/author team,
> but it is important for the reviewer to recognize that they may not
> result in change.  
> 
>> And part of the problem is to avoid turning it into a fight.  If all
>> review comments get clear, reasonably timely responses, there is room
>> for the discussion without acrimony.
> 
> I don't think that helps unless there is a clear set of expectations
> *on the part of reviewers* on what their responsibilities are in
> producing an actionable review and in accepting that some of their
> comments may result in no change.  I believe that the right way
> to do that is to ground the description of the review process in
> a strong understanding of the document production and participation
> model of the IETF.  If you don't, you end up with an unbalanced
> view in which the few hours of effort put in on a solicited review
> have a very much larger effect than the effort of the participants
> who produced the document and will use the protocol it
> describes.
> 
>> Would it address your concern if the document said something like:
>>    "Reviewers should be sensitive to the difference between
>>    their personal opinions (and preferences) and issues
>>    which will affect the correct operation or interoperation
>>    of the documents under review"
>> ?
> 
> I don't think this is nearly enough.
> 
> 					Ted
> 
> 
>> I have no problem with pointing out that there are two different
>> categories.  I have real problems with trying to define a hard line
>> which distinguishes them.
>>
>> Yours,
>> Joel
>>
>> PS: While there are other differences in our views, they seem to be
>> questions on which reasonable folk may differ and we can let the
>> community sort out how to write the wording.
>> _______________________________________________
>> IETF mailing list
>> IETF@ietf.org
>> https://www.ietf.org/mailman/listinfo/ietf
> 
> 
_______________________________________________
IETF mailing list
IETF@ietf.org
https://www.ietf.org/mailman/listinfo/ietf