[Icar] Re: [mpowr] Re: [Solutions] Summary of Discussion on Reforming IETF Quality Control Process

On Sat, 10 Jan 2004, Margaret Wasserman wrote:

> These concerns are what lead me to originally prefer the type of
> per-area review board that Alex Zinin has suggested, rather than a
> single large (SIRS-like) review board as you suggest.  I believe
> that it would be possible to give single ADs the ability to approve
> non-critical documents (criteria TBD) based on reviews by all of the
> per-area boards

I see no essential difference between a single board with members that
specialize in certain areas and multiple boards with members
specializing in board-specific area. From the point of view of the
important criteria you have been talking about, both designs are
essentially the same, IMO. I am not sure what concerns make one design
more attractive than another (on this level of abstraction). I suspect
the devil is in the details instead of high-level architectures.

For example, "what reviews are sufficient to publish a document" is a
question that can be answered equally well/poor with both single and
multiple boards structures.

If we concentrate on cross-area review and possibly lack volunteers, a
single board/pool sounds like a more natural framework.

> I think that we can somewhat consider the structure/management of
> the review board(s) to be orthogonal to whether some documents can
> be approved without full IESG review.

I agree.

> SCALABILITY
> ===========
>
> Like the SIRS proposal, I believe that this proposal is quite
> naive about the scale of this particular problem.

The draft-list-quality-control scalability depends on how many people
the ADs can approve as "certified reviewers" without approving bad
reviewers. I do not see how that process is essentially different from
any other process where there is a "trusted by final Authority" flag
assigned to a reviewer in one way or another.

If that flag is required for review consistency, then all approaches
would scale equally well/badly. It is not a problem we can "solve" by
using a process. We can only eliminate the scalability problem if we
remove the flag and, hence, require IESG to trust at least a certain
portion of unknown-to-IESG reviewers.

For example, we can say that any document that has no review conflicts
and has at least K reviews from each Area, with at least N reviews
from "trusted by AD/IESG" reviewers is published without IESG
involvement. Then you get a natural trade-off between scalability
(number of trusted reviewers required) and consistency/quality
(probability of IESG last-minute killing a document without a prior
review conflict and probability of IETF publishing a bad document
without a review conflict).

> CONSISTENCY
> ===========
>
> There are two types of consistency that are important for IETF
> document reviews:
>
>      (1) Consistency across the different reviews done for the
>          same document (to avoid thrash).

I do not understand what that means. Can you elaborate please?
Consistency in reviewer opinions during a round of reviews?
Consistency in reviewer opinions during the document lifetime?

>      (2) Using a consistent set of acceptance criteria for each
>          document that is reviewed/approved.

This is important indeed.

> Type (1) consistency could be achieved by having the same set of
> reviewers (to whatever extent possible) perform all of the
> reviews for a given document.

This can be achieved in any proposal I have seen, including
draft-list-quality-control-00, right?

> Type (2) consistency is _much_ harder to achieve across a large
> board (~200 people in my example above) than it is with a group of
> 13 people who spend several hours per week on the phone with each
> other, hold retreats and communicate daily...

I do not see a difference. If you can define acceptance criteria well,
then 10, 100, or 1000 people can use them with good-enough
consistency. If your acceptance criteria are so vague that they can
only exist as an undocumented belief system of 13 close friends
(different from the belief system of the other 13 friends), then we
probably do not need that kind of consistency.

> In order to achieve even a reasonable level of consistency across
> a large group (200 people?), I believe that we would need most or
> all of the following things:
>
>      - Written review criteria for each type of document.
>      - Documented per-area review criteria (like the MIB nits) for
>        every area or technical sub-area.
>      - A record of architectural/policy decisions that were made
>        that can be searched and used by subsequent reviewers.
>      - A mandatory preparation/training program (in-person or
>        on-line) for all new members of the review board.
>      - Some type of mentoring or monitoring program for new
>        members during the first NN reviews (most easily
>        achieved, perhaps, by having some sort of structure
>        within the board?).
>
> Without a plan (including committed resources) to achieve these
> things, I believe that a large review board would devolve into
> chaos.

How is N*13  reviewers groups would be any different?

I suspect there is either violation of the energy preservation laws in
these arguments or you are comparing apples to oranges.  The amount of
IETF review work that needs to be done should not change depending on
whether you use pre-area boards or one single pool. More precisely, if
the amount of work changes, then we are not comparing apples to apples
and we should decide on what the review system should accomplish
first.

> There may also be an issue with documents that are passed to the
> IESG for final approval.  The IESG's review may not be consistent
> with earlier reviews, perhaps resulting in "late surprises" or
> demotivation of the WG.  This problem is most likely in a situation
> where the IESG has no influence over what reviewers are chosen to do
> the initial review, and where the IESG does not feel accountable for
> the quality/correctness of the initial review (see section on
> accountability below).

I assumed that consistency with IESG decisions is included in (2). If
it is not, I do not understand your classification scheme, but the
laws of physics should still apply.

> CROSS-AREA COVERAGE
> ===================
>
> this proposal doesn't seem to do much to support or
> facilitate cross-area review.

I agree. The proposal requires an AD to ensure that all areas are
covered. This is a poor design because AD is responsible for one area
only.

We should find a scheme where all areas are represented when it comes
to cross-area review. For example, each area can have a list of
"trusted area reviewers" (all members of the IETF review board/pool).
And at least one(?) review must come from each area. This will ensure
that all areas are represented without relying on one area AD to take
care of that.

> Per-area review boards have a major advantage in this area.  If
> a document is passed by every per-area review board (particularly
> if those boards are carefully chosen to cover their area well
> and provide coverage for inter-area gaps), with the reviewers
> chosen by the ADs or other area review board leaders who aren't
> active in the WG, then it is more likely that a document will
> have received adequate cross-area review than if it is reviewed
> by a set of reviewers chosen from a large pool by the WG chair.

Indeed. However, what you describe as a "major advantage" can be
implemented equally well with a single pool design. Each reviewer has
a set of tags, including flags that mark that reviewer as a trusted
reviewer by an IETF Area Directorate. In fact, I can argue that a
single pool would work marginally better in some cases because Chairs
can easily find reviewers that are trusted by several areas and,
hence, reduce the number of reviews required. Common pool and
interface makes ensuring process rules easier as well.

> EFFICIENCY
> ==========
>
> This proposal might achieve this goal for some documents,
> particularly those that do not require IESG review.  However, the
> most important documents might be subjected to two rounds of
> review/approval -- one by the Quality Review Board and another by
> the IESG.  This issue might be resolved by better defining how/when
> a decision is made regarding whether a document will be
> reviewed/approved by the Quality Review Board or the IESG.

I agree and proposed some specific interfaces to solve this problem
(to the extent it can be solved when IESG authority differs from the
Board authority).

> MANAGEABILITY
> =============
>
> In order to run an orderly review process, to assign reviewers
> with appropriate (combinations of) expertise, to adquately
> training reviewers and to maintain the quality of the review
> process, the Quality Review Board would need to be managed.
> This proposal does not indicate how the board would be managed,
> who would manage it, how they would be chosen, etc.  This is
> a major omission that makes it difficult to evaluate many
> aspects of this proposal.

I tend to agree. However, I do note that the amount of required
management can be drastically reduced if we adopt a self-tuning open
pool of reviewers instead of going with a "traditional" managed
board(s) approach so typical to other SDOs and other non-volunteer
organizations.

> The per-area review board proposal relies on the ADs to select
> reviewers and manage the process as appropriate to each area.
> The AD would be responsible for ensuring that members of the
> area boards are adequately trained and prepared, and would
> be held accountable for the quality of their work.

... which has exactly the same manageability problems and a cross-area
consistency problem added on top. IMO, IETF areas are large enough to
be comparable (in scale) with IETF as a whole when it comes to
manageability

> ACCOUNTABILITY
> ==============
>
> This document seems to be based on the concept that quality
> review is performed for the WGs, and that the review process
> should be accountable to the WGs.  Ultimately, reviewers report
> to WG chairs, because they if they are not asked (by WG chairs)
> to review at least 3 document per year, they are removed.
> I believe that is wrong...

It would be wrong, but I disagree with your summary. While it is not
explicitly documented, the Review Board essentially works for and
accountable to IETF as a whole (just like a WG does/is). It should be
explicitly documented.

There are some minor mechanisms bugs (like requiring the Chair to
enter comments) that may create an impression that reviewers report to
WG chairs. I already commented on those; they can be fixed.

N.B. idle reviewers are _not_ removed. Only those that idle but also
rejecting review requests are.

> Another advantage of the per-area review board proposal over this
> proposal is that it builds upon, rather than replacing, the work
> that has already been done to build per-area review teams such as
> the MIB doctors, the ops-dir, Transport doctors, the rtg-dir, etc.

Members of those teams should be encouraged to volunteer to join the
open pool of reviewers, of course. We should not replace them, but
provide better access to and visibility for them!

Alex.

_______________________________________________
Icar mailing list
Icar@ietf.org
https://www1.ietf.org/mailman/listinfo/icar