Re: Telechat reviews [Re: Tooling glitch in Last Call announcements and records]

[Jean Mahoney <[e-mail]>]

John,

On 10/10/24 5:22 PM, John C Klensin wrote:
> Jean,
> 
> Per Brian, moving this to the IETF list and adjusting the subject
> line.  And pruning considerable text that I think was included in
> Brian's note and my response...
> 
> --On Thursday, October 10, 2024 14:54 -0500 Jean Mahoney
> <[e-mail]> wrote:
> 
>> (With my Gen-ART Secretary hat on)
>>
>> John,
>>
>> On 10/10/24 2:06 PM, John C Klensin wrote:
>>>
>>>
>>> --On Thursday, October 10, 2024 13:23 -0500 Robert Sparks
>>> <[e-mail]> wrote:
>> ...
>>> At least in principle, there is a difference between (i) Last Call
>>> as a community discussion mechanism whose effect is to inform the
>>> IESG about community consensus and (ii) Last Call as a mechanism
>>> to feed information, opinions, and other advice into the IESG so
>>> the ADs can determine what they think is the right decision for
>>> the Internet.  If those directorate/area reviews are given
>>> privileged status -- input into the telechats that ordinary IETF
>>> participants don't get, more flexibility about deadlines, etc.
>   
>> [JM] WRT to Gen-ART reviews, the reviewer should submit the review
>> before the Last Call.
> 
> Unclear.  Do you mean "before the Last Call starts and is therefore
> only a review for discussion within the area" or "before the Last
> Call ends".  

[JM] Before Last Call ends.

> If the former, I think that is a great idea -- it might
> even inform relevant ADs as to whether to initiate the Last Call.  I
> don't think that, in practice, that has been happening very often
> (certainly for draft-emailcore-rfc5321bis there has been no
> discussion on the ART list since well before publication was
> requested and, of course, no Gen-ART review posted at all so far.
> If the latter, that document constitutes a counterexample and, again,
> no posted review yet.

[JM] draft-ietf-emailcore-rfc5321bis is a long document (114 pages), and 
its Last Call deadline was the default two weeks after the announcement 
of Last Call. You can talk with the AD about extending the LC. This may 
help with receiving more reviews.

> 
>>   The telechat review that Robert mentioned is
>> when the Gen-ART reviewer follows up on their LC review (using the
>> same mailing lists that were used for the LC review) to say whether
>> their comments have/haven't been addressed.
> 
> But that requires that there be an earlier, public, review
> identifying those comments (inconsistent with "assigned at IETF Last
> Call" [1]).  In a way, it would constitute a supplement to the
> portion of the Shepherd's report that identifies outstanding issues.
> And, if it were what the IESG and community intended, the area
> reviews should probably be due, not during the Last Call window but a
> few days later so the reviewers can consider all Last Call comments
> and whether they were addressed.
> 
> If the reviews are assigned only when, or after, IETF Last Call
> starts, 

[JM] Yes, this is the case.

> then there presumably need to be two postings from the
> reviewer during the Last Call window -- the initial review with any
> issues identified and a second one, providing answers to the
> "addressed/not addressed" topics.

[JM] If the review highlights issues beyond nits, then it could prompt a 
discussion thread with the authors (note that Gen-ART reviews are sent 
to [e-mail] and the draft's WG mailing list if applicable in 
addition to the gen-art mailing list). These discussions can extend 
beyond the LC deadline.

> My entirely subjective
> impression is that almost never happens, at least in public and on
> the Last Call mailing list.
> 
>> I am currently not
>> assigning explicit telechat reviews because usually the reviewer
>> will follow up on their own.
> 
> Even, to come back to Brian's comment, less public.

[JM] The reviewer follows up on the lists to which they sent the LC 
review -- gen-art with [e-mail] and any relevant WG mailing 
list CCed, so the followups are public.

> 
>>> -- then the "treat this like any
>>> other review" boilerplate of most of those reviews becomes a joke
>>> or worse.  It would be somewhat different if those really were
>>> directorate or area reviews -- reviews that were written (or
>>> finalized) only after specific discussion about the document within
>>> that area or directorate and that represented consensus in that
>>> group.  But they often are not -- they are more often the opinions
>>> of an individual who comes up in rotation or draws a short straw.
> 
>> [JM] I assign a Gen-ART review to the next reviewer in rotation.
>> Please see [1] for details about the review team.
> 
> Nothing there surprises me, but, unless the reviewer reads the
> document, prepares a draft review, and posts it to an Area mailing
> list  (probably not just the review team list) for comment, it isn't
> really an Area review but a review from an individual who is assumed
> to have some of the perspective of the area.  Maybe that is happening
> in the General Area (or at least Gen-ART), 

[JM] Gen-ART reviews are from individuals who are reviewing documents 
from a general perspective. They consider the document's clarity, 
protocol architecture, normative language, normative references, and 
IANA Considerations when reviewing the document.

> but I have not seen
> symptoms of any multistage review of that type in any of the Areas I
> watch more closely.

[JM] Directorates can have different processes. Links to those processes 
can be found on their Datatracker pages [2].
> 
> In the same context, the problem with sharing draft reviews only with
> the Area review team 

[JM] Gen-ART reviewers don't create draft reviews for internal team 
discussion. A Gen-ART reviewer posts their review simultaneously to 
gen-art [3], [e-mail], and any relevant WG mailing list. The 
discussions are between the reviewer and the authors, and also with 
other WG participants and/or the AD, depending on the review.


Best regards,
Jean


> is that it means, unless special arrangements
> are made, one has to have the time to do reviews in order to see what
> reviews are going out in the Area's name.  For those who cannot spend
> unlimited time on the IETF or who have to make tradeoffs between
> general (or other area) work and their specific technical tasks, that
> is a hard problem -- indeed, making sure that documents are broadly
> reviewed from many perspectives, are what Last Calls are supposed to
> be about and the approach you describe might actually frustrate that.
> 
> best,
>      john
> 
>> [1] https://datatracker.ietf.org/group/genart/about/

[2] https://datatracker.ietf.org/dir/
[3] https://mailarchive.ietf.org/arch/browse/gen-art/