[Icar] Re: [Solutions] Summary of Discussion on Reforming IETF Quality Control Process

[Alex Rousskov <[e-mail]>]

On Fri, 9 Jan 2004, James Kempf wrote:

> I've written a draft summarizing the discussions on reforming the
> IETF quality control process that started on mpowr in December and
> wandered to solutions in January. It's here:
>
> http://www.geocities.com/kempf42/draft-list-quality-control-00.txt

James,

	Thank you for writing the draft. Here is my high-level
feedback. I assume we are not interested in low-level polishing at
this time.

1. "The ultimate responsibility for ensuring the quality
   of technical work done in the IETF lies with the Working Group..."

   I suspect you meant:

   "The ultimate responsibility for producing quality technical work
    in the IETF lies with the Working Group..."

   It is correct that WG/authors are responsible for producing quality
   output. It is wrong to imply that WG can ensure that. We need
   outside controls to ensure quality and your wording does not
   reflect that intent.

   Also, how does the above statement is affected by the fact that
   IESG can make WG-unapproved changes to WG documents? Do such
   changes make IESG responsible? Would it be more accurate to
   say that WG is responsible only up to a point where IESG takes
   control and does its conflict resolution thing?

2. "The ultimate responsibility for the quality of editorial content
   of the document describing the work lies with the authors and
   editors of the document."

   Sounds nice, but how does this reconcile with #1 above? The Author
   cannot overrule WG consensus. Are you implying that the author
   should resign if she disagrees with the WG consensus regarding
   her document?

3. "The IESG is responsible for deciding which Working Groups will
   require full IESG review on their documents"

   Wrong granularity, IMO. IESG decision should be made on document
   basis not WG basis, in general. That does not prevent IESG from
   reviewing every document of a given WG, of course, but it does
   not require that either.

4. "For documents that require IESG review or IETF management actions
   involving quality issues, the Area Director is responsible for
   ensuring that ...
   the [IESG] results are communicated to the Working Group, in as
   prompt a fashion as possible."

   A good review mechanism should not require manual involvement to
   communicate IESG (or any reviewer) actions on a document. We should
   avoid problems that manual forwarding of decisions always causes.

5. "Members of the board (called "Reviewers") may be
    subject experts with deep but fairly narrow expertise in one
    particular area, but more often they will have broad experience in
    a variety of areas involving Internet technology."

   I do not understand why deep expertise is worse than broad one.
   Why are you making this distinction? Is this sentence important?

6. Quality Review Board membership.

   I believe that any Board membership approach based on formal
   AD approval and/or popularity contest (IETF voting) is inferior
   to an "open pool of reviewers" approach. I hate to see IETF
   using such rigid mechanisms of "traditional" SDOs in a what is
   supposed to be an open volunteer organization.

   More specifically, I believe that the review pool/board should
   be open to anybody who volunteers and plays by the rules.

   There has to be a mechanism that identifies reviewers
   currently "trusted" by IESG, of course. Such mechanism
   will ensure consistency of reviews and IESG actions should the
   latter be necessary. Designing this mechanism is not trivial,
   but we can succeed if we focus on that rather than "membership
   rules".

   There should be no automatic inclusion or exclusion from the
   pool/board. We should not drag senior IETFers into the pool
   just because we can identify them by their IESG memberships
   or RFC counts. If they want to help, they will volunteer.
   We should not exclude idle reviewers from the pool because
   they did nothing wrong per se (rejecting review opportunities
   should not be counted _against_ anybody; it should be counted
   for solicitors to see and interpret).


7. "A determination of whether a full IESG review is necessary should
   be included in the Working Group Charter in order that the review
   process can proceed predictably"

   I do not understand the justification of this heavy requirement.
   Why is it important to know a priori that all WG documents will
   be fully reviewed by IESG? Will it make the WG work harder? Will
   it make the Chair look more important? Will it make other WG
   feel less important? What is the value to be gained from knowing
   that all WG documents will be reviewed by IESG, especially since
   IESG can review any document without a prior warning? This design
   also assumes that we are good at predicting what needs a
   full review even before the work has started.

   What problem does this WG flag solve??

8. Review Process

   The AD should not remove reviewers from the proposed list. The
   only thing an AD can do is to add reviewers if he thinks that
   proposed list does not provide enough trusted coverage.

   The review request should be sent to all reviewers accepting
   such requests, not just the ones on the AD-approved list.

   The whole design should make unknown but good reviewers known.
   This feature is essential for attracting always new/fresh
   reviewers and ensuring smooth scale.

   Full IESG review should not be a special case as far as
   notification and "handling" is concerned.

   All reviews must be archived and cross-referenced with
   documents and reviewers.


9. Disposition of Review Results

   The reviewers should be responsible for classifying their
   own comments and entering them into the review database.
   The Chair should not be involved at that stage. Again, we should
   not create a forwarding dependency/role when none is required.

   The above suggestion does create some extra work for some
   reviewers, but it is "good" work. It keeps reviews cleaner and
   more constructive. It also avoids problems with Chairs missing
   a comment or mis-classifying it. Finally, it actually saves
   reviewers time in the long run because they can check how their
   comments were addressed without mapping their comments with
   Chair's rendering of their comments.

   The review database/interface should be provided on IETF web
   site for everybody to use. The "issues page" is generated by
   IETF software, not Chairs. This provides everybody with a
   familiar interface and, again, solves "forwarding" problems
   like a forgotten, misrepresented, or even
   changed-after-final-reviewer-check issue on a Chair-controlled
   web page.

   IMO, we need an IETF-wide review management system to facilitate
   key review-related improvements we want.

Thank you,

Alex.

_______________________________________________
Icar mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/icar