Return-Path: X-Original-To: ietf@ietfa.amsl.com Delivered-To: ietf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A7307C1CAE77; Wed, 16 Oct 2024 09:42:29 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.653 X-Spam-Level: X-Spam-Status: No, score=-1.653 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.001, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01] autolearn=no autolearn_force=no Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TjPeQVoRKEwE; Wed, 16 Oct 2024 09:42:28 -0700 (PDT) Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CF13EC19ECB7; Wed, 16 Oct 2024 09:42:28 -0700 (PDT) Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-288dfdaf833so1041337fac.3; Wed, 16 Oct 2024 09:42:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729096948; x=1729701748; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xaMgUTbx4IRj2yjCSebO1+s6pTeyXTNpo/7aoFUh8TU=; b=OpQKj11SkpZ7rO5PIEY/XDdGnA06/klQqA5A1bqUUyrRyY2fEcaXrPhUOkY11/ck0w En3XimL4sw81i9XxHoM0YCrMEW4ONLqiZSL976IC7L86Ypp/Nm1olBf2PEE/lmDCabVK B+SWx9K7+jZYtC0mCt1hEWltZ30KmpqKYxXvybAWkULlfLXKhJg4/EUJjP4rPBCrAImM /tR/N4dI6FlcfPq8zm080eQAm+YrLLSBhKBk6m47KD5HOQyts5cLbaak03qQGEoxpF+M KoQMqQz6zWxzoKK4320bDwQHJdzu7IQ2o/knCiqiZSZvuBHHzJFRii+p0TG3cFjtiyP8 1uEw== X-Forwarded-Encrypted: i=1; AJvYcCUPukwfU7MCKmytu7YavaiY1IsNDrRYrk/biUgCRILPSXj2OORiAihtg5btoT7zn8//WOA7@ietf.org, AJvYcCW6iP7Hv9AMP7eHkR3suXd1RmAotB6UcGp5bw0GmmZ6tMdlr+QDaj581Ckf7ykAq0AiCNB/xg==@ietf.org X-Gm-Message-State: AOJu0Yxl0DCb+eZsHf3XycUVtGWlvjoKS+Tv9SmWdy4A+XdcUUjfUR7I GUSIjlwJbM4gtzoeMBUm8CA/7EVANMFjVpCZmZkC2+LUziBh41AQdTJrNIDWpDTDZBuc1z9cOaX nV6xpG/crhPnMJbBCjggu0r39Zaw= X-Google-Smtp-Source: AGHT+IGdjSrbEG5P/ASbOMHv4+rSgOXQKvslnpKFUXnhzulwvnrfUucWpLj0ywG2iKe5Wt7EblRCkIABOnz/bhiMSzs= X-Received: by 2002:a05:6870:972b:b0:267:dfce:95eb with SMTP id 586e51a60fabf-288eddb63b2mr3833563fac.5.1729096948085; Wed, 16 Oct 2024 09:42:28 -0700 (PDT) MIME-Version: 1.0 References: <822159B0D390905C0A194997@PSB> In-Reply-To: From: Barry Leiba Date: Wed, 16 Oct 2024 12:42:16 -0400 Message-ID: Subject: Re: Telechat reviews [Re: Tooling glitch in Last Call announcements and records] To: "Rob Wilton (rwilton)" Content-Type: multipart/alternative; boundary="0000000000005f7e0a06249ac13d" Message-ID-Hash: PQEK6JI6CDJ6MEZXRURGL3PAPMSWAZCA X-Message-ID-Hash: PQEK6JI6CDJ6MEZXRURGL3PAPMSWAZCA X-MailFrom: barryleiba@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-ietf.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: John C Klensin , "iesg@ietf.org" , "ietf@ietf.org" X-Mailman-Version: 3.3.9rc6 Precedence: list List-Id: "IETF-Discussion. This is the most general IETF mailing list, intended for discussion of technical, procedural, operational, and other topics for which no dedicated mailing lists exist." Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --0000000000005f7e0a06249ac13d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Working group chairs can always request early reviews at any time =E2=80=94= they seldom do, but they sometimes do, particularly when they know a document is complex or needs someone with specific expertise to have a look. Reviews are automatically requested by the tooling when the document enters the Last Call state (and the IESG Evaluation state with a telechat date set). That=E2=80=99s where we get most of the review requests, exactly bec= ause it=E2=80=99s automatic. No one has to think about it and (remember to) ask= . We could certainly use other state transitions (such as =E2=80=9CIn WG Last= Call=E2=80=9D) to trigger an automatic request, but I don=E2=80=99t think that would work = well in general. Some working groups have multiple =E2=80=9Clast calls=E2=80=9D fo= r various reasons, and I think chairs would rather not have side effects such as this. Perhaps a separate check-box on state changes for =E2=80=9Crequest directorate and review team reviews=E2=80=9D would work. In general, anything that requires a specific request will mostly not be used. Barry On Wed, Oct 16, 2024 at 12:32=E2=80=AFPM Rob Wilton (rwilton) wrote: > Hi Barry, > > Another choice, that perhaps could be considered, would be to initiate th= e > directorate reviews slightly earlier in the cycle. E.g., at the point th= at > the WG has said that is ready before publication but before the AD has > reviewed and agreed to publish. In fact, input from the directorate > reviews might be very helpful input to decide whether the document is > really ready to progress, or if there are significant issues outstanding. > > > > Of course, this might mean that a second follow up lighter directorate > review is needed to cover any changes that occurred between the initial > review and the version going before the IESG ballot, but if that second > review was focussed on the differences and issues raised previously then = I > would have thought that the increase in workload on the directorate would > probably be fairly small, and hopefully manageable. I.e., I am assuming > that the second review would be assigned back to the originate directorat= e > reviewer. > > Generally, I think that it is better to get as many reviews as early as > possible in the process when the folks working on the document and still > very fresh and vested in getting the document published. Perhaps bigger > changes to the process could also be considered =E2=80=A6 > > > > Anyway, just a thought. > > > Regards, > Rob > > > > > > *From: *Barry Leiba > *Date: *Wednesday, 16 October 2024 at 13:47 > *To: *Murray S. Kucherawy > *Cc: *John C Klensin , iesg@ietf.org , > ietf@ietf.org > *Subject: *Re: Telechat reviews [Re: Tooling glitch in Last Call > announcements and records] > > We should make it a general policy to add two weeks to the last call > period when a document is long, for some value of "long" (I might say > over 60 pages of substance (not counting change logs and such)). I > try to get to assigning ART-ART reviews a couple of times a week, but > that still means that, depending upon the timing, with a two-week last > call I might be giving a reviewer only a 7- or 8-day deadline for a > 100+-page document, and I always blanch when I have to do that. While > ADs regularly have to review long documents with a week or two notice, > I think it's unreasonable to expect last-call reviews from > directorates/review-teams on that notice for long documents. > > We decided on the two-week last call period at a different time, when > the IETF was a different organization. Maybe we should re-think it > now, and keep in mind that an extra two weeks of last-call review is > *not* going to be the most significant delay in a document's life > cycle. > > Barry, ART-ART manager > > On Fri, Oct 11, 2024 at 5:53=E2=80=AFPM Murray S. Kucherawy > wrote: > > > > Hi John, > > > > On Fri, Oct 11, 2024 at 2:01=E2=80=AFPM John C Klensin > wrote: > >> > >> Thanks for the clarification. Seems entirely reasonable with one or > >> two qualifications. First, if you (and/or other areas) are doing > >> things that way, the review needs to be posted to the Last Call list > >> well before the Last Call closes out so there is time for people from > >> the Area and the broader community to comment on it. Second, if the > >> posted end of Last Call date is unreasonable or unattainable for some > >> reason, I'd hope the responsible AD could be notified of that early > >> in the Last Call window -- at least no later than a week before it is > >> closed -- rather than, e.g., after the close date. That would permit > >> actions, if needed, to be taken without things looking like a game of > >> "Gotcha" with the AD and WG and/or author(s) responsible for the > >> document. > > > > > > For what it's worth, in my time on the IESG, I haven't found the need t= o > manage this vigorously. If there's a directorate review I'd really like = to > have, I have the discretion to wait for it before scheduling the document > onto a telechat even though Last Call has ended. If the review has come = in > but it provokes discussion, I have the discretion to wait for that > discussion to resolve before moving forward. If we're talking about a > document that isn't one of mine and a review comes in from my area review > team raising something on which I'd like to dive deeper, I can use DISCUS= S > for that (so long as I am diligent about clearing it once the discussion = is > had, of course). That's been my strategy for a while now and it's never > raised a complaint, which (so far, at least) includes the document you're > talking about here. > > > > The thing I used to determine if the review has come in is the > datatracker. I will check the last-call list too, but the datatracker > provides a nice snapshot of which reviews have been requested and which > have come in, and is usually where I start when checking on a document's > status. > > > > Just to keep this all public: For this particular document, I have > pinged the assigned directorate reviewers to ask them to upload their > reviews ASAP on this document. As I said elsewhere, I might be fine > advancing a document missing a couple of directorate reviews, but not all > of them. If they don't come in soon, I'll reach out to the review team > chairs to ask for reassignments. > > > > Lastly, I would definitely appreciate a notification (automated or > otherwise) when a directorate review is going to be late. Right now all > the tracker tells me is "not done", which could mean "not done yet" or > could mean "don't hold your breath". > > > > -MSK > --0000000000005f7e0a06249ac13d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Working group chairs can always request early reviews at = any time =E2=80=94 they seldom do, but they sometimes do, particularly when= they know a document is complex or needs someone with specific expertise t= o have a look.

Reviews a= re automatically requested by the tooling when the document enters the Last= Call state (and the IESG Evaluation state with a telechat date set).=C2=A0= That=E2=80=99s where we get most of the review requests, exactly because i= t=E2=80=99s automatic.=C2=A0 No one has to think about it and (remember to)= ask.

We could certainly= use other state transitions (such as =E2=80=9CIn WG Last Call=E2=80=9D) to= trigger an automatic request, but I don=E2=80=99t think that would work we= ll in general.=C2=A0 Some working groups have multiple =E2=80=9Clast calls= =E2=80=9D for various reasons, and I think chairs would rather not have sid= e effects such as this.=C2=A0 Perhaps a separate check-box on state changes= for =E2=80=9Crequest directorate and review team reviews=E2=80=9D would wo= rk.

In general, anything= that requires a specific request will mostly not be used.

Barry

On Wed, Oct 16, 2024 at 12:32= =E2=80=AFPM Rob Wilton (rwilton) <r= wilton@cisco.com> wrote:

Hi Barry,

Another choice, that perhaps could be considered, would be to initiate the = directorate reviews slightly earlier in the cycle.=C2=A0 E.g., at the point= that the WG has said that is ready before publication but before the AD ha= s reviewed and agreed to publish.=C2=A0 In fact, input from the directorate reviews might be very helpful input to de= cide whether the document is really ready to progress, or if there are sign= ificant issues outstanding.

=C2=A0

Of course, this mig= ht mean that a second follow up lighter directorate review is needed to cov= er any changes that occurred between the initial review and the version goi= ng before the IESG ballot, but if that second review was focussed on the differences= and issues raised previously then I would have thought that the increase i= n workload on the directorate would probably be fairly small, and hopefully= manageable.=C2=A0 I.e., I am assuming that the second review would be assigned back to the originate directorate= reviewer.

Generally, I think = that it is better to get as many reviews as early as possible in the proces= s when the folks working on the document and still very fresh and vested in= getting the document published.=C2=A0 Perhaps bigger changes to the process could = also be considered =E2=80=A6

=C2=A0

Anyway, just a thou= ght.=C2=A0


Regards,
Rob



=C2=A0

From: Barry Leiba <barryleiba@computer.org>
Date: Wednesday, 16 October 2024 at 13:47
To: Murray S. Kucherawy <superuser@gmail.com>
Cc: John C Klensin <john-ietf@jck.com>, iesg@ietf.org <iesg@ietf.org>, ietf@ietf.org <ietf@ietf.org>
Subject: Re: Telechat reviews [Re: Tooling glitch in Last Call annou= ncements and records]

We should make it a general policy to add = two weeks to the last call
period when a document is long, for some value of "long" (I might= say
over 60 pages of substance (not counting change logs and such)).=C2=A0 I try to get to assigning ART-ART reviews a couple of times a week, but
that still means that, depending upon the timing, with a two-week last
call I might be giving a reviewer only a 7- or 8-day deadline for a
100+-page document, and I always blanch when I have to do that.=C2=A0 While=
ADs regularly have to review long documents with a week or two notice,
I think it's unreasonable to expect last-call reviews from
directorates/review-teams on that notice for long documents.

We decided on the two-week last call period at a different time, when
the IETF was a different organization.=C2=A0 Maybe we should re-think it now, and keep in mind that an extra two weeks of last-call review is
*not* going to be the most significant delay in a document's life
cycle.

Barry, ART-ART manager

On Fri, Oct 11, 2024 at 5:53=E2=80=AFPM Murray S. Kucherawy <superuser@gmail.com> wrote:
>
> Hi John,
>
> On Fri, Oct 11, 2024 at 2:01=E2=80=AFPM John C Klensin <john-ietf@jck.com> wrote:
>>
>> Thanks for the clarification.=C2=A0 Seems entirely reasonable with= one or
>> two qualifications.=C2=A0 First, if you (and/or other areas) are d= oing
>> things that way, the review needs to be posted to the Last Call li= st
>> well before the Last Call closes out so there is time for people f= rom
>> the Area and the broader community to comment on it.=C2=A0=C2=A0 S= econd, if the
>> posted end of Last Call date is unreasonable or unattainable for s= ome
>> reason, I'd hope the responsible AD could be notified of that = early
>> in the Last Call window -- at least no later than a week before it= is
>> closed -- rather than, e.g., after the close date.=C2=A0 That woul= d permit
>> actions, if needed, to be taken without things looking like a game= of
>> "Gotcha" with the AD and WG and/or author(s) responsible= for the
>> document.
>
>
> For what it's worth, in my time on the IESG, I haven't found t= he need to manage this vigorously.=C2=A0 If there's a directorate revie= w I'd really like to have, I have the discretion to wait for it before = scheduling the document onto a telechat even though Last Call has ended.=C2=A0 If the review has come in but it provokes discussion= , I have the discretion to wait for that discussion to resolve before movin= g forward.=C2=A0 If we're talking about a document that isn't one o= f mine and a review comes in from my area review team raising something on which I'd like to dive deeper, I can use DISCUSS = for that (so long as I am diligent about clearing it once the discussion is= had, of course).=C2=A0 That's been my strategy for a while now and it&= #39;s never raised a complaint, which (so far, at least) includes the document you're talking about here.
>
> The thing I used to determine if the review has come in is the datatra= cker.=C2=A0 I will check the last-call list too, but the datatracker provid= es a nice snapshot of which reviews have been requested and which have come= in, and is usually where I start when checking on a document's status.
>
> Just to keep this all public: For this particular document, I have pin= ged the assigned directorate reviewers to ask them to upload their reviews = ASAP on this document.=C2=A0 As I said elsewhere, I might be fine advancing= a document missing a couple of directorate reviews, but not all of them.=C2=A0 If they don't come in soon, I'= ll reach out to the review team chairs to ask for reassignments.
>
> Lastly, I would definitely appreciate a notification (automated or oth= erwise) when a directorate review is going to be late.=C2=A0 Right now all = the tracker tells me is "not done", which could mean "not do= ne yet" or could mean "don't hold your breath".
>
> -MSK

--0000000000005f7e0a06249ac13d--