Re: Telechat reviews [Re: Tooling glitch in Last Call announcements and records]

[Jean Mahoney <[e-mail]>]

John,

A further clarification -- the last-call mailing list is included by 
default when a Gen-ART reviewer sends their review through Datatracker:
https://datatracker.ietf.org/mailtrigger/name/review_completed_genart_lc/

Best regards,
Jean

On 10/11/24 3:59 PM, John C Klensin wrote:
> Jean,
> 
> Thanks for the clarification.  Seems entirely reasonable with one or
> two qualifications.  First, if you (and/or other areas) are doing
> things that way, the review needs to be posted to the Last Call list
> well before the Last Call closes out so there is time for people from
> the Area and the broader community to comment on it.   Second, if the
> posted end of Last Call date is unreasonable or unattainable for some
> reason, I'd hope the responsible AD could be notified of that early
> in the Last Call window -- at least no later than a week before it is
> closed -- rather than, e.g., after the close date.  That would permit
> actions, if needed, to be taken without things looking like a game of
> "Gotcha" with the AD and WG and/or author(s) responsible for the
> document.
> 
> thanks and have a good weekend.
>      john
> 
> 
> --On Friday, October 11, 2024 13:28 -0500 Jean Mahoney
> <[e-mail]> wrote:
> 
>> John,
>>
>> On 10/10/24 5:22 PM, John C Klensin wrote:
>>> Jean,
>>>
>>> Per Brian, moving this to the IETF list and adjusting the subject
>>> line.  And pruning considerable text that I think was included in
>>> Brian's note and my response...
>>>
>>> --On Thursday, October 10, 2024 14:54 -0500 Jean Mahoney
>>> <[e-mail]> wrote:
>>>
>>>> (With my Gen-ART Secretary hat on)
>>>>
>>>> John,
>>>>
>>>> On 10/10/24 2:06 PM, John C Klensin wrote:
>>>>>
>>>>>
>>>>> --On Thursday, October 10, 2024 13:23 -0500 Robert Sparks
>>>>> <[e-mail]> wrote:
>>>> ...
>>>>> At least in principle, there is a difference between (i) Last
>>>>> Call as a community discussion mechanism whose effect is to
>>>>> inform the IESG about community consensus and (ii) Last Call as
>>>>> a mechanism to feed information, opinions, and other advice into
>>>>> the IESG so the ADs can determine what they think is the right
>>>>> decision for the Internet.  If those directorate/area reviews
>>>>> are given privileged status -- input into the telechats that
>>>>> ordinary IETF participants don't get, more flexibility about
>>>>> deadlines, etc.
>>>    
>>>> [JM] WRT to Gen-ART reviews, the reviewer should submit the review
>>>> before the Last Call.
>>>
>>> Unclear.  Do you mean "before the Last Call starts and is therefore
>>> only a review for discussion within the area" or "before the Last
>>> Call ends".
>>
>> [JM] Before Last Call ends.
>>
>>> If the former, I think that is a great idea -- it might
>>> even inform relevant ADs as to whether to initiate the Last Call.
>>> I don't think that, in practice, that has been happening very often
>>> (certainly for draft-emailcore-rfc5321bis there has been no
>>> discussion on the ART list since well before publication was
>>> requested and, of course, no Gen-ART review posted at all so far.
>>> If the latter, that document constitutes a counterexample and,
>>> again, no posted review yet.
>>
>> [JM] draft-ietf-emailcore-rfc5321bis is a long document (114
>> pages), and its Last Call deadline was the default two weeks after
>> the announcement of Last Call. You can talk with the AD about
>> extending the LC. This may help with receiving more reviews.
>>
>>>
>>>>    The telechat review that Robert mentioned is
>>>> when the Gen-ART reviewer follows up on their LC review (using the
>>>> same mailing lists that were used for the LC review) to say
>>>> whether their comments have/haven't been addressed.
>>>
>>> But that requires that there be an earlier, public, review
>>> identifying those comments (inconsistent with "assigned at IETF
>>> Last Call" [1]).  In a way, it would constitute a supplement to the
>>> portion of the Shepherd's report that identifies outstanding
>>> issues. And, if it were what the IESG and community intended, the
>>> area reviews should probably be due, not during the Last Call
>>> window but a few days later so the reviewers can consider all Last
>>> Call comments and whether they were addressed.
>>>
>>> If the reviews are assigned only when, or after, IETF Last Call
>>> starts,
>>
>> [JM] Yes, this is the case.
>>
>>> then there presumably need to be two postings from the
>>> reviewer during the Last Call window -- the initial review with any
>>> issues identified and a second one, providing answers to the
>>> "addressed/not addressed" topics.
>>
>> [JM] If the review highlights issues beyond nits, then it could
>> prompt a discussion thread with the authors (note that Gen-ART
>> reviews are sent to [e-mail] and the draft's WG mailing
>> list if applicable in addition to the gen-art mailing list). These
>> discussions can extend beyond the LC deadline.
>>
>>> My entirely subjective
>>> impression is that almost never happens, at least in public and on
>>> the Last Call mailing list.
>>>
>>>> I am currently not
>>>> assigning explicit telechat reviews because usually the reviewer
>>>> will follow up on their own.
>>>
>>> Even, to come back to Brian's comment, less public.
>>
>> [JM] The reviewer follows up on the lists to which they sent the LC
>> review -- gen-art with [e-mail] and any relevant WG
>> mailing list CCed, so the followups are public.
>>
>>>
>>>>> -- then the "treat this like any
>>>>> other review" boilerplate of most of those reviews becomes a joke
>>>>> or worse.  It would be somewhat different if those really were
>>>>> directorate or area reviews -- reviews that were written (or
>>>>> finalized) only after specific discussion about the document
>>>>> within that area or directorate and that represented consensus
>>>>> in that group.  But they often are not -- they are more often
>>>>> the opinions of an individual who comes up in rotation or draws
>>>>> a short straw.
>>>
>>>> [JM] I assign a Gen-ART review to the next reviewer in rotation.
>>>> Please see [1] for details about the review team.
>>>
>>> Nothing there surprises me, but, unless the reviewer reads the
>>> document, prepares a draft review, and posts it to an Area mailing
>>> list  (probably not just the review team list) for comment, it
>>> isn't really an Area review but a review from an individual who is
>>> assumed to have some of the perspective of the area.  Maybe that
>>> is happening in the General Area (or at least Gen-ART),
>>
>> [JM] Gen-ART reviews are from individuals who are reviewing
>> documents from a general perspective. They consider the document's
>> clarity, protocol architecture, normative language, normative
>> references, and IANA Considerations when reviewing the document.
>>
>>> but I have not seen
>>> symptoms of any multistage review of that type in any of the Areas
>>> I watch more closely.
>>
>> [JM] Directorates can have different processes. Links to those
>> processes can be found on their Datatracker pages [2].
>>>
>>> In the same context, the problem with sharing draft reviews only
>>> with the Area review team
>>
>> [JM] Gen-ART reviewers don't create draft reviews for internal team
>> discussion. A Gen-ART reviewer posts their review simultaneously to
>> gen-art [3], [e-mail], and any relevant WG mailing list.
>> The discussions are between the reviewer and the authors, and also
>> with other WG participants and/or the AD, depending on the review.
>>
>>
>> Best regards,
>> Jean
>>
>>
>>> is that it means, unless special arrangements
>>> are made, one has to have the time to do reviews in order to see
>>> what reviews are going out in the Area's name.  For those who
>>> cannot spend unlimited time on the IETF or who have to make
>>> tradeoffs between general (or other area) work and their specific
>>> technical tasks, that is a hard problem -- indeed, making sure
>>> that documents are broadly reviewed from many perspectives, are
>>> what Last Calls are supposed to be about and the approach you
>>> describe might actually frustrate that.
>>>
>>> best,
>>>       john
>>>
>>>> [1] https://datatracker.ietf.org/group/genart/about/
>>
>> [2] https://datatracker.ietf.org/dir/
>> [3] https://mailarchive.ietf.org/arch/browse/gen-art/
>>
> 
>