Re: [Icar] independence of reviews; variability

Although Dave and Harald are both using the term "review" in this 
thread, I think that the they may be talking about two very different 
things...  In my mind, I would call these things review and 
evaluation.

IMO, review is a purely informative activity.  The reviewer reads the 
document and provides information about its quality, clarity, 
feasibility, etc.  A review generally includes both positive and 
negative feedback.  The WG can then use that review to inform their 
decisions about the document -- whether to accept it as a WG work 
item, how to modify it over time, whether or not to submit it for 
publication.  The reviewer is not making a decision about the 
document, but is instead providing information to the WG that can be 
used to make a decision.  I agree with Dave that this activity should 
be independent of the IESG/IAB leadership and that this can best be 
accomplished as a widely-distributed grass roots function.

There is also another concept that I will call "evaluation".  In 
today's process, there are two evaluations done on every IETF 
document:  an AD evaluation and an IESG evaluation.  These 
evaluations involves reading the document and _deciding_ whether or 
not it is suitable for publication.  In general, the AD and the IESG 
do not provide reviews (as I used the term above) as part of this 
evaluation process.  If we decide that a document is not suitable for 
publication in its current form, we do tell the WG why, but we do not 
attempt (nor, IMO, should we) to provide a complete, balanced review 
of the document.

IMO, both review and evaluation are necessary parts of the document 
process, but they don't need to be done by the same people, nor are 
they done at the same time in the process.

There are a number of things that I believe that this WG could do to 
support, encourage and improve reviews within the IETF, including:

    - Defining a mechanism for how WG chairs/editors can get reviewers assigned
      to their documents (perhaps building from experience with MIB 
Doctors, SIRs, etc.)

    - Setting guidelines for the type and quantity of review that 
should be obtained
      at each stage in the WG process (before accepting as WG work item, before
      sending to WG last call, before submitting for publication, etc.).

    - Establish review criteria for each stage in the WG process, 
perhaps including
      some type of review forms.

    - Provide input to the EDU team, so that we can educate IETF 
participants on how
      to provide useful and productive reviews.

This group might also look at how we can make better use of reviews 
to inform and accelerate the evaluation process.

Margaret

_______________________________________________
Icar mailing list
Icar@ietf.org
https://www1.ietf.org/mailman/listinfo/icar