Re: Quality of Directorate reviews

Michael Richardson <mcr+ietf@sandelman.ca> Sat, 16 November 2019 06:42 UTC

Return-Path: <mcr@sandelman.ca>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C59131200D5 for <ietf@ietfa.amsl.com>; Fri, 15 Nov 2019 22:42:27 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.435
X-Spam-Level: *
X-Spam-Status: No, score=1.435 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_SBL_CSS=3.335, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CLXmhkIzcwDf for <ietf@ietfa.amsl.com>; Fri, 15 Nov 2019 22:42:26 -0800 (PST)
Received: from relay.sandelman.ca (relay.cooperix.net [IPv6:2a01:7e00::f03c:91ff:feae:de77]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7ED8B1200CE for <ietf@ietf.org>; Fri, 15 Nov 2019 22:42:26 -0800 (PST)
Received: from dooku.sandelman.ca (unknown [IPv6:2001:67c:370:128:2472:ebff:feda:c977]) by relay.sandelman.ca (Postfix) with ESMTPS id DAB9D1F451; Sat, 16 Nov 2019 06:42:24 +0000 (UTC)
Received: by dooku.sandelman.ca (Postfix, from userid 179) id DAC81BF7; Sat, 16 Nov 2019 14:42:20 +0800 (CST)
From: Michael Richardson <mcr+ietf@sandelman.ca>
To: Keith Moore <moore@network-heretics.com>
cc: ietf@ietf.org
Subject: Re: Quality of Directorate reviews
In-reply-to: <84ee7053-1dbb-bfcc-c576-c2cf115a743e@network-heretics.com>
References: <157279399807.13506.13363770981495597049.idtracker@ietfa.amsl.com> <0EF64763-BA25-468A-B387-91445A61D318@gmail.com> <CAJU8_nUovmFmgNiYx0ez_1f+GPdU9xGViDYWfowEEomrn0pyDw@mail.gmail.com> <alpine.LRH.2.21.1911040841160.27600@bofh.nohats.ca> <CE06CC6D-E37F-4C90-B782-D14B1D715D4B@cable.comcast.com> <38E47448-63B4-4A5D-8A9D-3AB890EBDDDD@akamai.com> <09886edb-4302-b309-9eaa-f016c4487128@gmail.com> <26819.1572990657@localhost> <2668fa45-7667-51a6-7cb6-4b704c7fba5a@isode.com> <2C97D18E-3DA0-4A2D-8179-6D86EB835783@gmail.com> <MN2PR11MB43669E4CEF13CDA51A764F9AB5790@MN2PR11MB4366.namprd11.prod.outlook.com> <20191.1573054128@localhost> <15BCDF05-FB13-45D2-A5DF-70618EBA1A5A@gmail.com> <9182.1573147520@localhost> <A3493C65-7F8A-407D-A9F4-FF36296C0920@gmail.com> <CAMm+LwiP4Ypuyh2xsd8qBjUfwuNzOYOfp3OrDnPmU-YwMH2pMw@mail.gmail.com> <02eb79d1-1830-5830-ed95-b743f601a8de@network-heretics.com> <f60f410e-1cab-368b-b981-4e85c0f6a816@sandelman.ca> <84ee7053-1dbb-bfcc-c576-c2cf115a743e@network-he retics.com>
Comments: In-reply-to Keith Moore <moore@network-heretics.com> message dated "Fri, 15 Nov 2019 09:08:50 -0500."
X-Mailer: MH-E 8.6; nmh 1.6; GNU Emacs 24.5.1
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-="; micalg="pgp-sha256"; protocol="application/pgp-signature"
Date: Sat, 16 Nov 2019 14:42:20 +0800
Message-ID: <31471.1573886540@dooku.sandelman.ca>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/3btSs7Gkpwihq9HzYSekW-kDQc8>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Nov 2019 06:42:28 -0000

Keith Moore <moore@network-heretics.com> wrote:
    >> On 2019-11-13 11:25 p.m., Keith Moore wrote:
    >>> On 11/13/19 10:07 AM, Phillip Hallam-Baker wrote:
    >>>
    >>>> Maybe what we need is a structure that assigns multiple reviewers
    >>>> for some projects and rubber stamps others.
    >>> Seems like ADs already have a fair amount of discretion to ask for
    >>> multiple in-depth reviewers vs. getting minimal review.   If having a
    >>> human make such decisions isn't your idea of an appropriate
    >>> "structure", I'd be curious to know what is.
    >>>
    >> The issue is that is only so much senior security clue to go around.
    >> There is a non-trivial amount of effort for an-out-area reviewer to
    >> spin up enough understanding about what a WG is doing.  There are a
    >> lot of documents that simply allocate a new attribute from an existing
    >> registry and then use it for something.  Determining if this has a
    >> trivial or non-trivial security impact can be difficult.  If it turns
    >> out to be trivial, then we've wasted the reviewers time (opportunity
    >> cost).  If it turns out not to be trivial (and the reviewer missed
    >> that), then if we are lucky, we catch it at IESG time, and then it
    >> might be a year later.

    > I don't disagree with any of the above.  And yet, I don't see how it's
    > responding to either of the above replies.

The current system assigns the review prior to the AD determining if they
need an in-depth review or not.  So if we assign a senior (security) reviewer
to a document that didn't need in-depth senior experience, then that person
is unavailable (within the quantum of review assignment period) for the AD to
assign them to do something more in-depth.

--
Michael Richardson <mcr+IETF@sandelman.ca>, Sandelman Software Works
 -= IPv6 IoT consulting =-