Re: [Rfcplusplus] Conversation as metaphor

[Peter Saint-Andre <[e-mail]>]

On 7/11/18 10:19 AM, RFC ISE (Adrian Farrel) wrote:
> Peter Saint-Andre wrote:
> 
>> In what sense are Independent Stream documents peer reviewed?
>>
>> At a higher level, is there a good definition of "peer review" so that
>> we can understand how documents from the various streams map to that
>> definition?
> 
> Peter, "what is a peer and what constitutes a peer review?" are questions
> that have bothered me for a while even for tier one journals. I have
> received some frankly appalling journal reviews over the years (and I say
> that not just because I have disagreed with them) where the reviewers have
> clearly not understood the fundamentals of the material. I suspect we have
> all read journal article where we have wondered about the source of the
> substance being smoked.
> 
> So you might be asking: what review is performed on the Independent
> Submissions stream? There is no secret about that, so here is a summary in
> no particular order:
> 
> - The ISE reviews. This might not be a subject matter expert review as the
> ISE is a generalist with some areas of detailed knowledge. The review is
> often more for consistency and readability.
> 
> - The ISE commissions two or three reviews from persons suggested by the
> authors but who are not directly involved in the work. It is evident to
> the ISE from these reviews whether the reviewer has sufficient subject
> knowledge to be classed as a peer. Note that failure to get sufficient
> reviews in this class is gating on publication - if even the authors can't
> find people to review their work then it really is of limited interest.
> 
> - The ISE requests reviews from other experts in the IETF community. These
> are often drawn from the ISEB (names available on the web), but may be
> people that the ISE can see have a close interest in the subject matter.
> (If you would like to be more involved in this sort of review then I will
> bite your hand off!)
> 
> - The RFC 5742 review by the IESG often draws review comments, although
> the IESG is not required or expected to provide such a review.

Thanks for the clear summary of process and criteria.

> PS. Some I-Ds pass through WG and IETF last call with no comments at all
> (possibly suggesting no one is reviewing them) and get relatively light
> review in the IESG. Saying that is not an attack - just a comparison.

Most definitely.

We as a community seem to be confused about what level of "peer review"
we think is important, appropriate, necessary, or sufficient. We're
engineers, not academics, so comparing our publications to academic
journals might not be relevant.

Peter