Re: Review Board Scalability (was: Re: [Icar] Input based on SIRS experience)

On Mon, 12 Jan 2004, James Kempf wrote:
> It seems to me that, in principle, the only difference between an IETF
> review board and the program committee for a conference is that the members
> of an IETF review board would be expected to stick with a draft for more
> than just one review, and possibly in the number of reviewers per paper. So
> it seems like we might be able to look at existing experience with
> conference program committees for some idea about how to make this work.

Note that there may be some unstated assumptions here.

For example, that such an IETF Review Board would replace some 
functions of the IESG, such as:
 - AD Evaluation
 - Review for RFC-Editor submissions
 - Review of WG submission of Info/Experimental documents
 - Review of individual submissions of --""--
 - Review of standards track documents as well

If you assume that (or all of these), you will undoubtedly arrive at a 
conclusion that any proposal will lack manpower.

However, if the idea is to improve cross-area or cross-functional 
review so that some of the above will go *easier*, requiring much 
smaller amount of time, we might be getting somewhere.

So, when proposing a review board(s), it's useful to explicitly state 
which functions we want to replace (or augment).

> If the review board's work is just advisory, as the ART and SIRS proposals
> do, then I think we will have a hard time getting people to volunteer.

I'm not sure if this is necessarily the case.  Speaking from personal
experience, there is several kinds of responses a reviewer will not
want to see:

- "it's too late to make reviews [especially on this specific issue
[which may have had consensus at X]], but thanks anyway", "

- (comments are received, but they're not taken into account)
  * at least in a timely fashion (following up within days is easier 
    than after months)

> Imagine a program committee for a conference in which the reviews produced
> by the program committee could be overridden by the program's organizing
> committee, 

Organizing committee would not be a problem, at least to me -- you can 
consider it as "they know best" -- at least after discussion.

> or even by the authors (this would be the equivalent of the IESG
> or the WGs themselves, respectively, being allowed to override a review
> board opinion). Who would volunteer?

The authors and WGs are a different thing entirely.  If they failed to
see a problem in the first place, I would not count on them to judge 
whether a problem needs to solved or not.

> There is another kind of conference program committee in which the authors
> themselves are required to do reviews. This could be another way to solicit
> reviews, people who are draft editors are required to provide reviews.

Heh, an interesting idea :-)

Along the same lines, you could also require that each WG chair must
make N reviews every year.  An instant base of 200+ reviewers :-)

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

_______________________________________________
Icar mailing list
Icar@ietf.org
https://www1.ietf.org/mailman/listinfo/icar