Review Board Scalability (was: Re: [Icar] Input based on SIRS experience)

Margaret Wasserman said:

>Like the SIRS proposal, I believe that this proposal is quite
>naive about the scale of this particular problem.  Let's take
>a few numbers:
>
>We currently approve about 200 RFCs per year.  Each of these
>RFCs receives (on average) ~2-1/2 review cycles from the IESG
>plus a full AD Review.  So, let's assume that we will continue to
>produce 200 documents per year, and that each will be subjected
>to 3 cross-area review cycles.
>
>BTW, if the same group of people reviews the document all three
>times (also see section on consistency below), the later reviews
>will take much less time than the earlier reviews.
>
>The IESG consists of 13 people, not all of whom carefully review
>each document during IESG review (for various reason).  So, let's
>assume that we can get adequate cross-area coverage (see section
>on cross-area coverage) by having each document reviewed by
>8 properly selected members of the Quality Review Board.  That's
>the number of ADs that it currently takes (today, with one slot
>empty) to approve publication of a document.
>
>So, the number of individual document reviews required would be
>200 * 3 * 8 == 4800.
>
>Let's assume that we can find 200 people who are willing and
>qualified to serve on the Quality Review Board.  I don't know
>that this is possible, and it means that someone will have to
>manage a function that involves 200 people (see manageability
>section below), but let's assume...  In that case, each member
>of the review board would need to do an average of 24 reviews
>per year -- so a minimum of 3 is misleadingly low.  Ideally,
>this means that each member of the review board would do a
>full 3-cycle review for 8 documents per year.
>
>If we expect this system to result in a 50% improvement in
>document throughput, we will need to handle 300 documents per
>year, which requires 36 reviews/board member/year (or 12
>documents).
>
>Doable?  Maybe.  If we can find 200 people willing to do this,
>figure out how to train and organize them (see sections on
>preparation/training and management below), and sustain that
>number over time.  The sign-up rate for SIRS does not make me
>confident that this is possible, but we could try....

and Spencer Dawkins said:

>  Yeah, Margaret is also coming up with 200-300 reviewers needed, and
>  we  still have no idea how many we can get. :-{

It seems to me that, in principle, the only difference between an IETF
review board and the program committee for a conference is that the members
of an IETF review board would be expected to stick with a draft for more
than just one review, and possibly in the number of reviewers per paper. So
it seems like we might be able to look at existing experience with
conference program committees for some idea about how to make this work.

As an example, I'm on the program committee for Mobihoc this year. There are
42 program committee members and 230 papers which works out to a ratio of
about 6 papers per reviewer. I was assigned 11 papers, from which I conclude
that the program chairs have decided to have about 2 reviews per paper.

If one accepts Margaret's numbers of about 24 reviews per year, that would
imply that the number of reviews for IETF drafts would be about twice what
Mobihoc members are expected to do. This doesn't seem all that bad to me
considering that I'll probably do at least 11 additional reviews this year,
of IETF drafts and papers for other conferences, etc. Working backwards from
the Mobihoc program committee, that would reduce the number of review board
members to maybe 90 (twice the Mobihoc number) rather than 200.

To do this, however, we would have to reduce the number of independent
_solicited_ reviewers to 2 rather than Margaret's suggestion of 8. Would
this cause a reduction of quality detection problems? Unclear. Many drafts
now receive unsolicited review as part of WG last call and IETF last call
and presumably that would not change. Margaret's number of 8 reviews was
based on an approximation of how many ADs review drafts, so YMMV. And, even
with 8 people in the IESG reviewing, there have been occasions when problems
have been caught later (MIPv6 binding update security comes to mind). Also,
with an issue tracking system in place, the reviewers may be able to
determine whether issues are resolved more easily without having to wade
through the text in the spec, so reviews might become less time consuming.

I think there is a different issue that is possibly more of a barrier,
getting people to volunteer by providing the right incentive. People
volunteer for a conference program commitee because they want to see what
other people in the area are up to from a reseach standpoint, and they want
to make sure that published results are of high quaility. The program
committee's comments are used to determine what papers are ultimately
published, that is, their reviews make a real difference in the conference
content.

If the review board's work is just advisory, as the ART and SIRS proposals
do, then I think we will have a hard time getting people to volunteer.
Imagine a program committee for a conference in which the reviews produced
by the program committee could be overridden by the program's organizing
committee, or even by the authors (this would be the equivalent of the IESG
or the WGs themselves, respectively, being allowed to override a review
board opinion). Who would volunteer?

There is another kind of conference program committee in which the authors
themselves are required to do reviews. This could be another way to solicit
reviews, people who are draft editors are required to provide reviews.
Obviously, since draft editors are less experienced, their reviews could not
be as highly rated as the review board's reviews. In such conferences, I
presume the organizing committee has the final say, and typically these
kinds of conferences are not first tier.

            jak

_______________________________________________
Icar mailing list
Icar@ietf.org
https://www1.ietf.org/mailman/listinfo/icar