Re: Quality of Directorate reviews

[Michael Richardson <[e-mail]>]

Benjamin Kaduk <[e-mail]> wrote:
    >> Keith Moore <[e-mail]> wrote: >> On 2019-11-13 11:25
    >> p.m., Keith Moore wrote: >>> On 11/13/19 10:07 AM, Phillip
    >> Hallam-Baker wrote:

    >> >>>> Maybe what we need is a structure that assigns multiple reviewers
    >> >>>> for some projects and rubber stamps others.  >>> Seems like ADs
    >> already have a fair amount of discretion to ask for >>> multiple
    >> in-depth reviewers vs. getting minimal review.   If having a >>> human
    >> make such decisions isn't your idea of an appropriate >>> "structure",
    >> I'd be curious to know what is.

    >> >> The issue is that is only so much senior security clue to go
    >> around.  >> There is a non-trivial amount of effort for an-out-area
    >> reviewer to >> spin up enough understanding about what a WG is doing. 
    >> There are a >> lot of documents that simply allocate a new attribute
    >> from an existing >> registry and then use it for something. 
    >> Determining if this has a >> trivial or non-trivial security impact
    >> can be difficult.  If it turns >> out to be trivial, then we've wasted
    >> the reviewers time (opportunity >> cost).  If it turns out not to be
    >> trivial (and the reviewer missed >> that), then if we are lucky, we
    >> catch it at IESG time, and then it >> might be a year later.
    >>
    >> > I don't disagree with any of the above.  And yet, I don't see how
    >> it's > responding to either of the above replies.
    >>
    >> The current system assigns the review prior to the AD determining if
    >> they need an in-depth review or not.  So if we assign a senior
    >> (security) reviewer to a document that didn't need in-depth senior
    >> experience, then that person is unavailable (within the quantum of
    >> review assignment period) for the AD to assign them to do something
    >> more in-depth.

    > My understanding is that most directorates have a secretary that does
    > the assignments (secdir does, at least).

yes, that's my understanding.

I'd like to see more coordination between ADs (particularly Sec-ADs) and
directorates so that the security review process can occur earlier, and so
that any loop with the SecADs can happen earlier.

In the case of draft-ietf-anima-bootstrapping-keyinfra, I'd have liked to get
more attention from Christian,Jari and Russ (reviewers) and the various ADs
earlier.  The significant reviews were done a year ago, and we are just
finishing now.
That's a big investment of time among the 6 or 7 people involved.

    > By the time an AD is looking
    > at the review next to the document it might only be a few days before
    > the telechat where the document is up for approval, which is not really
    > enough time to get another review in without deferring the document.

It seems that we doing these early secdir reviews, but someone this is not
feeding up to the ADs well enough, who then do their own review.  That's just
not leveraging the secdir well.

    > Maybe we should go get that extra review and try to remove the stigma
    > against deferring documents; I don't have a sense for how the community
    > would feel about that.

I'm okay with this, but maybe the sponsoring AD and WG chairs need to be more
active in chasing down reviewers.

Again, I'd like more offocial acknowledgement of the work reviewers do.

    > And yes, the AD should look at the directorate review when it arrives,
    > but looking only at the review and not the document being reviewed is
    > not always enough to tell whether additional review would be valuable.

Agreed.
What if the Shepherd write up was had more ways to flag things?

--
Michael Richardson <[e-mail]>, Sandelman Software Works
 -= IPv6 IoT consulting =-