Re: Quality of Directorate reviews

Benjamin Kaduk <kaduk@mit.edu> Sat, 16 November 2019 07:08 UTC

Return-Path: <kaduk@mit.edu>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6034C1200E3 for <ietf@ietfa.amsl.com>; Fri, 15 Nov 2019 23:08:10 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.2
X-Spam-Level:
X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WoRoevpRiiXL for <ietf@ietfa.amsl.com>; Fri, 15 Nov 2019 23:08:07 -0800 (PST)
Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A6B1912008D for <ietf@ietf.org>; Fri, 15 Nov 2019 23:08:07 -0800 (PST)
Received: from kduck.mit.edu ([24.16.140.251]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id xAG782Kj016369 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 16 Nov 2019 02:08:05 -0500
Date: Fri, 15 Nov 2019 23:08:02 -0800
From: Benjamin Kaduk <kaduk@mit.edu>
To: Michael Richardson <mcr+ietf@sandelman.ca>
Cc: Keith Moore <moore@network-heretics.com>, ietf@ietf.org
Subject: Re: Quality of Directorate reviews
Message-ID: <20191116070802.GN32847@kduck.mit.edu>
References: <MN2PR11MB43669E4CEF13CDA51A764F9AB5790@MN2PR11MB4366.namprd11.prod.outlook.com> <20191.1573054128@localhost> <15BCDF05-FB13-45D2-A5DF-70618EBA1A5A@gmail.com> <9182.1573147520@localhost> <A3493C65-7F8A-407D-A9F4-FF36296C0920@gmail.com> <CAMm+LwiP4Ypuyh2xsd8qBjUfwuNzOYOfp3OrDnPmU-YwMH2pMw@mail.gmail.com> <02eb79d1-1830-5830-ed95-b743f601a8de@network-heretics.com> <f60f410e-1cab-368b-b981-4e85c0f6a816@sandelman.ca> <84ee7053-1dbb-bfcc-c576-c2cf115a743e@network-heretics.com> <31471.1573886540@dooku.sandelman.ca>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <31471.1573886540@dooku.sandelman.ca>
User-Agent: Mutt/1.12.1 (2019-06-15)
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/s5z_83BQau4bDiXn5xhhakxyvYg>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Nov 2019 07:08:10 -0000

On Sat, Nov 16, 2019 at 02:42:20PM +0800, Michael Richardson wrote:
> 
> Keith Moore <moore@network-heretics.com> wrote:
>     >> On 2019-11-13 11:25 p.m., Keith Moore wrote:
>     >>> On 11/13/19 10:07 AM, Phillip Hallam-Baker wrote:
>     >>>
>     >>>> Maybe what we need is a structure that assigns multiple reviewers
>     >>>> for some projects and rubber stamps others.
>     >>> Seems like ADs already have a fair amount of discretion to ask for
>     >>> multiple in-depth reviewers vs. getting minimal review.   If having a
>     >>> human make such decisions isn't your idea of an appropriate
>     >>> "structure", I'd be curious to know what is.
>     >>>
>     >> The issue is that is only so much senior security clue to go around.
>     >> There is a non-trivial amount of effort for an-out-area reviewer to
>     >> spin up enough understanding about what a WG is doing.  There are a
>     >> lot of documents that simply allocate a new attribute from an existing
>     >> registry and then use it for something.  Determining if this has a
>     >> trivial or non-trivial security impact can be difficult.  If it turns
>     >> out to be trivial, then we've wasted the reviewers time (opportunity
>     >> cost).  If it turns out not to be trivial (and the reviewer missed
>     >> that), then if we are lucky, we catch it at IESG time, and then it
>     >> might be a year later.
> 
>     > I don't disagree with any of the above.  And yet, I don't see how it's
>     > responding to either of the above replies.
> 
> The current system assigns the review prior to the AD determining if they
> need an in-depth review or not.  So if we assign a senior (security) reviewer
> to a document that didn't need in-depth senior experience, then that person
> is unavailable (within the quantum of review assignment period) for the AD to
> assign them to do something more in-depth.

My understanding is that most directorates have a secretary that does the
assignments (secdir does, at least).  By the time an AD is looking at the
review next to the document it might only be a few days before the telechat
where the document is up for approval, which is not really enough time to
get another review in without deferring the document.  Maybe we should go
get that extra review and try to remove the stigma against deferring
documents; I don't have a sense for how the community would feel about
that.

And yes, the AD should look at the directorate review when it arrives, but
looking only at the review and not the document being reviewed is not
always enough to tell whether additional review would be valuable.

-Ben