Re: Telechat reviews [Re: Tooling glitch in Last Call announcements and records]

[John C Klensin <[e-mail]>]

Jean,

Thanks for the clarification.  Seems entirely reasonable with one or
two qualifications.  First, if you (and/or other areas) are doing
things that way, the review needs to be posted to the Last Call list
well before the Last Call closes out so there is time for people from
the Area and the broader community to comment on it.   Second, if the
posted end of Last Call date is unreasonable or unattainable for some
reason, I'd hope the responsible AD could be notified of that early
in the Last Call window -- at least no later than a week before it is
closed -- rather than, e.g., after the close date.  That would permit
actions, if needed, to be taken without things looking like a game of
"Gotcha" with the AD and WG and/or author(s) responsible for the
document.

thanks and have a good weekend.
    john


--On Friday, October 11, 2024 13:28 -0500 Jean Mahoney
<[e-mail]> wrote:

> John,
> 
> On 10/10/24 5:22 PM, John C Klensin wrote:
>> Jean,
>> 
>> Per Brian, moving this to the IETF list and adjusting the subject
>> line.  And pruning considerable text that I think was included in
>> Brian's note and my response...
>> 
>> --On Thursday, October 10, 2024 14:54 -0500 Jean Mahoney
>> <[e-mail]> wrote:
>> 
>>> (With my Gen-ART Secretary hat on)
>>> 
>>> John,
>>> 
>>> On 10/10/24 2:06 PM, John C Klensin wrote:
>>>> 
>>>> 
>>>> --On Thursday, October 10, 2024 13:23 -0500 Robert Sparks
>>>> <[e-mail]> wrote:
>>> ...
>>>> At least in principle, there is a difference between (i) Last
>>>> Call as a community discussion mechanism whose effect is to
>>>> inform the IESG about community consensus and (ii) Last Call as
>>>> a mechanism to feed information, opinions, and other advice into
>>>> the IESG so the ADs can determine what they think is the right
>>>> decision for the Internet.  If those directorate/area reviews
>>>> are given privileged status -- input into the telechats that
>>>> ordinary IETF participants don't get, more flexibility about
>>>> deadlines, etc.
>>   
>>> [JM] WRT to Gen-ART reviews, the reviewer should submit the review
>>> before the Last Call.
>> 
>> Unclear.  Do you mean "before the Last Call starts and is therefore
>> only a review for discussion within the area" or "before the Last
>> Call ends".  
> 
> [JM] Before Last Call ends.
> 
>> If the former, I think that is a great idea -- it might
>> even inform relevant ADs as to whether to initiate the Last Call.
>> I don't think that, in practice, that has been happening very often
>> (certainly for draft-emailcore-rfc5321bis there has been no
>> discussion on the ART list since well before publication was
>> requested and, of course, no Gen-ART review posted at all so far.
>> If the latter, that document constitutes a counterexample and,
>> again, no posted review yet.
> 
> [JM] draft-ietf-emailcore-rfc5321bis is a long document (114
> pages), and its Last Call deadline was the default two weeks after
> the announcement of Last Call. You can talk with the AD about
> extending the LC. This may help with receiving more reviews.
> 
>> 
>>>   The telechat review that Robert mentioned is
>>> when the Gen-ART reviewer follows up on their LC review (using the
>>> same mailing lists that were used for the LC review) to say
>>> whether their comments have/haven't been addressed.
>> 
>> But that requires that there be an earlier, public, review
>> identifying those comments (inconsistent with "assigned at IETF
>> Last Call" [1]).  In a way, it would constitute a supplement to the
>> portion of the Shepherd's report that identifies outstanding
>> issues. And, if it were what the IESG and community intended, the
>> area reviews should probably be due, not during the Last Call
>> window but a few days later so the reviewers can consider all Last
>> Call comments and whether they were addressed.
>> 
>> If the reviews are assigned only when, or after, IETF Last Call
>> starts, 
> 
> [JM] Yes, this is the case.
> 
>> then there presumably need to be two postings from the
>> reviewer during the Last Call window -- the initial review with any
>> issues identified and a second one, providing answers to the
>> "addressed/not addressed" topics.
> 
> [JM] If the review highlights issues beyond nits, then it could
> prompt a discussion thread with the authors (note that Gen-ART
> reviews are sent to [e-mail] and the draft's WG mailing
> list if applicable in addition to the gen-art mailing list). These
> discussions can extend beyond the LC deadline.
> 
>> My entirely subjective
>> impression is that almost never happens, at least in public and on
>> the Last Call mailing list.
>> 
>>> I am currently not
>>> assigning explicit telechat reviews because usually the reviewer
>>> will follow up on their own.
>> 
>> Even, to come back to Brian's comment, less public.
> 
> [JM] The reviewer follows up on the lists to which they sent the LC
> review -- gen-art with [e-mail] and any relevant WG
> mailing list CCed, so the followups are public.
> 
>> 
>>>> -- then the "treat this like any
>>>> other review" boilerplate of most of those reviews becomes a joke
>>>> or worse.  It would be somewhat different if those really were
>>>> directorate or area reviews -- reviews that were written (or
>>>> finalized) only after specific discussion about the document
>>>> within that area or directorate and that represented consensus
>>>> in that group.  But they often are not -- they are more often
>>>> the opinions of an individual who comes up in rotation or draws
>>>> a short straw.
>> 
>>> [JM] I assign a Gen-ART review to the next reviewer in rotation.
>>> Please see [1] for details about the review team.
>> 
>> Nothing there surprises me, but, unless the reviewer reads the
>> document, prepares a draft review, and posts it to an Area mailing
>> list  (probably not just the review team list) for comment, it
>> isn't really an Area review but a review from an individual who is
>> assumed to have some of the perspective of the area.  Maybe that
>> is happening in the General Area (or at least Gen-ART), 
> 
> [JM] Gen-ART reviews are from individuals who are reviewing
> documents from a general perspective. They consider the document's
> clarity, protocol architecture, normative language, normative
> references, and IANA Considerations when reviewing the document.
> 
>> but I have not seen
>> symptoms of any multistage review of that type in any of the Areas
>> I watch more closely.
> 
> [JM] Directorates can have different processes. Links to those
> processes can be found on their Datatracker pages [2].
>> 
>> In the same context, the problem with sharing draft reviews only
>> with the Area review team 
> 
> [JM] Gen-ART reviewers don't create draft reviews for internal team
> discussion. A Gen-ART reviewer posts their review simultaneously to
> gen-art [3], [e-mail], and any relevant WG mailing list.
> The discussions are between the reviewer and the authors, and also
> with other WG participants and/or the AD, depending on the review.
> 
> 
> Best regards,
> Jean
> 
> 
>> is that it means, unless special arrangements
>> are made, one has to have the time to do reviews in order to see
>> what reviews are going out in the Area's name.  For those who
>> cannot spend unlimited time on the IETF or who have to make
>> tradeoffs between general (or other area) work and their specific
>> technical tasks, that is a hard problem -- indeed, making sure
>> that documents are broadly reviewed from many perspectives, are
>> what Last Calls are supposed to be about and the approach you
>> describe might actually frustrate that.
>> 
>> best,
>>      john
>> 
>>> [1] https://datatracker.ietf.org/group/genart/about/
> 
> [2] https://datatracker.ietf.org/dir/
> [3] https://mailarchive.ietf.org/arch/browse/gen-art/
>