Re: Quality of Directorate reviews

Benjamin Kaduk <kaduk@mit.edu> Mon, 18 November 2019 10:53 UTC

Return-Path: <kaduk@mit.edu>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9FADB1208F2 for <ietf@ietfa.amsl.com>; Mon, 18 Nov 2019 02:53:09 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.2
X-Spam-Level:
X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2LQIgOVKiaPd for <ietf@ietfa.amsl.com>; Mon, 18 Nov 2019 02:53:08 -0800 (PST)
Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C8C781200DB for <ietf@ietf.org>; Mon, 18 Nov 2019 02:53:07 -0800 (PST)
Received: from mit.edu ([24.16.140.251]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id xAIAr3cW021143 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 18 Nov 2019 05:53:05 -0500
Date: Mon, 18 Nov 2019 02:53:03 -0800
From: Benjamin Kaduk <kaduk@mit.edu>
To: Michael Richardson <mcr+ietf@sandelman.ca>
Cc: ietf@ietf.org
Subject: Re: Quality of Directorate reviews
Message-ID: <20191118105303.GN32847@mit.edu>
References: <15BCDF05-FB13-45D2-A5DF-70618EBA1A5A@gmail.com> <9182.1573147520@localhost> <A3493C65-7F8A-407D-A9F4-FF36296C0920@gmail.com> <CAMm+LwiP4Ypuyh2xsd8qBjUfwuNzOYOfp3OrDnPmU-YwMH2pMw@mail.gmail.com> <02eb79d1-1830-5830-ed95-b743f601a8de@network-heretics.com> <f60f410e-1cab-368b-b981-4e85c0f6a816@sandelman.ca> <84ee7053-1dbb-bfcc-c576-c2cf115a743e@network-heretics.com> <31471.1573886540@dooku.sandelman.ca> <20191116070802.GN32847@kduck.mit.edu> <10449.1573959523@dooku.sandelman.ca>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <10449.1573959523@dooku.sandelman.ca>
User-Agent: Mutt/1.12.1 (2019-06-15)
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/ufFRcNPyjgfO3vuff_Lxd92BGoU>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Nov 2019 10:53:10 -0000

Hi Michael,

On Sun, Nov 17, 2019 at 10:58:43AM +0800, Michael Richardson wrote:
> 
> Benjamin Kaduk <kaduk@mit.edu>; wrote:
>     > My understanding is that most directorates have a secretary that does
>     > the assignments (secdir does, at least).
> 
> yes, that's my understanding.
> 
> I'd like to see more coordination between ADs (particularly Sec-ADs) and
> directorates so that the security review process can occur earlier, and so
> that any loop with the SecADs can happen earlier.
> 
> In the case of draft-ietf-anima-bootstrapping-keyinfra, I'd have liked to get
> more attention from Christian,Jari and Russ (reviewers) and the various ADs
> earlier.  The significant reviews were done a year ago, and we are just
> finishing now.
> That's a big investment of time among the 6 or 7 people involved.

I'm not sure that I understand what you're looking for here.
In the case of
https://datatracker.ietf.org/doc/review-ietf-anima-bootstrapping-keyinfra-16-secdir-lc-huitema-2018-09-29/,
my conclusion from looking at the review is that "this document is not
ready for publication as-is and should go back to the WG before it comes
before the IESG".  That is, I would essentially ignore the document until
the secdir reviewer is satisfied [or the authors' responses give some
indication that the reviewer is incorrect], to make more efficient use of
my time.  But it sounds like you're suggesting that I should see that
review and take it as a signal to get *more* involved with the document.
Am I missing something?

>     > By the time an AD is looking
>     > at the review next to the document it might only be a few days before
>     > the telechat where the document is up for approval, which is not really
>     > enough time to get another review in without deferring the document.
> 
> It seems that we doing these early secdir reviews, but someone this is not
> feeding up to the ADs well enough, who then do their own review.  That's just
> not leveraging the secdir well.

In a similar vein, I'm not sure what you would see as "leveraging the
secdir well".  I don't see myself as beholden to accept the secdir review
as-is -- to me the secdir reviews are a tool that I can take advantage of
as I perform my AD duties, but it's not the only tool at my disposal, and
if I am concerned that a given document may have broad impact or contain
subtle security issues, I am generally going to do an in-depth review
myself, in addition to any other reviews that have been done.

>     > Maybe we should go get that extra review and try to remove the stigma
>     > against deferring documents; I don't have a sense for how the community
>     > would feel about that.
> 
> I'm okay with this, but maybe the sponsoring AD and WG chairs need to be more
> active in chasing down reviewers.

Just to check: this is "seeking enough reviews from the relevant area(s)"
as opposed to "ensuring that people who did reviews respond to the updates
made because of their reviews"?

> Again, I'd like more offocial acknowledgement of the work reviewers do.
> 
>     > And yes, the AD should look at the directorate review when it arrives,
>     > but looking only at the review and not the document being reviewed is
>     > not always enough to tell whether additional review would be valuable.
> 
> Agreed.
> What if the Shepherd write up was had more ways to flag things?

>From a technical point of view, I think this may only be relevant when the
shepherd is not a WG chair for the WG in question -- my understanding is
that the WG chair can click a button in the datatracker to request a
directorate review at any time, which might be more helpful than just a
note in the writeup.

-Ben