Re: Quality of Directorate reviews

[Benjamin Kaduk <[e-mail]>]

On Sat, Nov 16, 2019 at 02:42:20PM +0800, Michael Richardson wrote:
> 
> Keith Moore <[e-mail]> wrote:
>     >> On 2019-11-13 11:25 p.m., Keith Moore wrote:
>     >>> On 11/13/19 10:07 AM, Phillip Hallam-Baker wrote:
>     >>>
>     >>>> Maybe what we need is a structure that assigns multiple reviewers
>     >>>> for some projects and rubber stamps others.
>     >>> Seems like ADs already have a fair amount of discretion to ask for
>     >>> multiple in-depth reviewers vs. getting minimal review.   If having a
>     >>> human make such decisions isn't your idea of an appropriate
>     >>> "structure", I'd be curious to know what is.
>     >>>
>     >> The issue is that is only so much senior security clue to go around.
>     >> There is a non-trivial amount of effort for an-out-area reviewer to
>     >> spin up enough understanding about what a WG is doing.  There are a
>     >> lot of documents that simply allocate a new attribute from an existing
>     >> registry and then use it for something.  Determining if this has a
>     >> trivial or non-trivial security impact can be difficult.  If it turns
>     >> out to be trivial, then we've wasted the reviewers time (opportunity
>     >> cost).  If it turns out not to be trivial (and the reviewer missed
>     >> that), then if we are lucky, we catch it at IESG time, and then it
>     >> might be a year later.
> 
>     > I don't disagree with any of the above.  And yet, I don't see how it's
>     > responding to either of the above replies.
> 
> The current system assigns the review prior to the AD determining if they
> need an in-depth review or not.  So if we assign a senior (security) reviewer
> to a document that didn't need in-depth senior experience, then that person
> is unavailable (within the quantum of review assignment period) for the AD to
> assign them to do something more in-depth.

My understanding is that most directorates have a secretary that does the
assignments (secdir does, at least).  By the time an AD is looking at the
review next to the document it might only be a few days before the telechat
where the document is up for approval, which is not really enough time to
get another review in without deferring the document.  Maybe we should go
get that extra review and try to remove the stigma against deferring
documents; I don't have a sense for how the community would feel about
that.

And yes, the AD should look at the directorate review when it arrives, but
looking only at the review and not the document being reviewed is not
always enough to tell whether additional review would be valuable.

-Ben