Re: Cross-area review (was Meeting rotation)

[John C Klensin <[e-mail]>]

One small addition...

--On Monday, December 21, 2015 16:39 -0800 Dave Crocker
<[e-mail]> wrote:

>       3. We have no discipline to reviews and no documentation
> to give careful guidance.
> 
>          Reviews are a random walk of well-intentioned people
> who have varying skills, varying attention-spans, etc.  The
> results are, therefore, random.

Perhaps worse, some of the area reviews are not random but are
from people who, when they are engaging in full disclosure,
admit that they didn't understand the spec and therefore
reviewed what they did understand and maybe engaged in a bit of
nit-picking to show that they really tried to read it.  I've
occasionally seen almost exactly the same behavior from a few
ADs.  Maybe more than "occasionally" depending on how critically
one evaluates the reviews.

Beyond a certain point, those types of reviews make us feel
good, but may make a negative contribution to document quality
by convincing others, who might have less time (or not draw the
short straw) and/or better expertise and insight to conclude
that they don't need to read the document because lots of others
are doing so.

> In other words, it is merely a certainty that serious problems
> will slip through.  Every review is good to do, but no
> specific review is the savior of us all.  (The ultimate
> fallacy of a savior model is requiring expenditure of scarce,
> strategic area director resources on late-stage reviews...
> Expensive resources, frustrating delays, minimal benefits. Who
> wouldn't want all that?)

Exactly.

> The main benefit of a cross-area review is that it is a cold
> reading by someone with no history with the effort.  Fresh
> eyes.

I would have said "no history with the effort and at least some
substantive knowledge and interest in the subject matter".   "I
don't have a clue but was designated as the Foobar Area
Reviewer" reviews are usually not very helpful (although
sometimes we get lucky).

>...
> We could move that requirement to Proposed, but that would
> merely make Proposed essentially the same as Full.  And the
> barrier to Proposed is already too high.
> 
> Instead we should understand that we cannot and should not try
> to demand or expect documents that are perfect.  We should
> demand 'good enough' and let the outside world evaluate and
> feed the results back to us.

I'm not certain we are getting to "good enough" quite often
enough.  Moreover, what was "good enough" when the expectation
was that people would not deploy Proposed Standards in products,
at least without understanding that was a risk and treating it
as such, may not be "good enough" when Proposed Standards are
not only deployed but the community's attitude seems to be that
a piece of specification that turns out to be a bad idea has to
be truly catastrophic to modify the spec in an incompatible way,

>...
> The problem is our mythology before that and the misguided
> expectations and barriers created as a result.

Very likely... at least a major part of the problem.

    john