Re: Post-Last-Call versions of documents and change logs: suggestion to the IESG

[Brian E Carpenter <[e-mail]>]

(Front posting to avoid a TL;DR.)

It's already the case that if the AD considers that the changes after Last Call and IESG review are substantive, a second Last Call can (and should) be issued. Isn't that sufficient? It does rely on the AD's judgment, of course, and should probably be done more often.

I agree about the change log, although I tend to rely on rfcdiff or iddiff.

Regards
    Brian Carpenter

On 26-Jun-22 04:49, John C Klensin wrote:
> IESG,
> 
> While neither issue is new, I've been troubled in the last few
> weeks by a pair of related issues.  I want to identify them (as
> briefly as I can) and, while it won't solve them, make some
> small practical suggestions.  I am writing about WG documents
> but, especially with various interpretations of RFC 8789,
> AD-sponsored individual ones may be even more subject to
> problems.
> 
> In the last few steps of our document approval process, an IETF
> Last Call is announced and announced about a specific version of
> a document.  If there are useful (typically as judged by the
> authors, not, e.g., the WG that is responsible for the document)
> comments during Last Call, the document is revised during and/or
> immediately after that period, sometimes more than once.  After
> the Last Call period ends, the IESG starts its formal review, in
> theory making a determination of community consensus based on
> the Last Call comments.    In practice, the ADs often find
> issues not identified during Last Call and their comments often
> lead to further document changes before a draft (possibly with
> notes) goes off to the RFC Editor.  Over the years, some of the
> issues caught in IESG review have been very important so the
> present model, as practiced, is is probably nearly as good as we
> can do.  However, it raises two very fundamental questions about
> IETF consensus (rough or otherwise) about the finished work:
> 
> (i) Is the document on which the IESG review starts
> substantively the same as the one on which the Last Call was
> started?  Would any of the changes have elicited addition
> substantive community comments had the been in the document when
> the Last Call started?  Presumably both questions are answered
> by the responsible AD in allowing the document to move into IESG
> evaluation, but that means one person, with an investment in the
> document and the WG, is making a potentially significant
> decision on behalf of the community.
> 
> (ii)  When changes are made to the document at IESG request (or
> even late discovery of issues by the authors), does the document
> forwarded to the RFC Editor still represent community consensus?
> Is that question even explicitly asked in all cases?
> 
> These two stages of the process are, at least in principle, even
> more important than the AUTH48 one that has drawn recent
> attention.   I don't think there is a "solution" to either, but
> have some suggestions to mitigate the possible risks of a
> document being published that does not represent community
> consensus.  AFAICT, none of them require revisions to normative
> process documents or rules.
> 
> (1) While it has never been a requirement (and, IMO, should not
> be), many WGs and authors have included "Change Logs" that
> summarize differences among versions of an I-D.  If new versions
> of an I-D are produced after the Last Call starts, the IESG
> should insist on such summaries if the modified versions are to
> be considered without restarting the Last Call using those
> versions.  That would allow community members who have already
> reviewed and commented on a previous version, are working on
> comments, and even those who have decided they do not need to
> post Last Call comments based on a earlier version to swiftly
> determine whether they need to reexamine the I-D or particular
> sections of it.  While an irresponsible or malicious editor
> could game that requirement in the hope of slipping something
> through, that risk is better dealt with by choices of editors
> (or firing them if needed) than by eliminating the benefits of
> responsible behavior.
> 
> (2) While there are many advantages of making changes to
> documents by Github pull requests, they do not make it easy for
> someone concerned about aspects of a document but without the
> time to get fully drawn into the work (IMO, exactly the people
> IETF Last call is intended to reach because those who are more
> involved presumably would have been part of the WG) to
> understand all changes in context.  Probably there should be a
> clear requirement that significant changes (e.g., replacement or
> additions of whole sections or appendices) be accompanied by a
> new draft so that it is possible to do a diff.  Even more
> important, if _any_ changes were made during Last Call (or
> during AD review before IESG review starts), a new version
> should be posted so that the community can be aware of exactly
> what the IESG is evaluating.  While I am less concerned about
> it, it would be good for the IESG to consider whether a new
> draft should be required along with a Protocol Action or
> Document Action notice rather than reflecting any but the most
> trivial of correction in IESG notes to the RPC.
> 
> (3) One way to mitigate some of the perceived problems would be
> for the IESG to be much more ready to push documents back to the
> originating WGs when significant issues are raised or changes
> made, requiring at least a brief WG Last Call.  That would
> assure that there is at least still WG consensus around the
> post-Last Call version of a document the WG presumably produced.
> It would obviously add an extra week or two to the process when
> things go smoothly and so should be used with discretion.  But
> it would, in some (many?) cases be faster and more consistent
> with fairness than an extended discussion or negotiation between
> authors and IESG members.  It would also be consistent with a
> view that, if significant changes are needed to a document after
> Last Call starts, they are often indicative of insufficient
> diligence on the part of the WG to be sure that necessary
> substantive issues have been reviewed and considered before
> publication is requested.
> 
> (4) AFAICT, current procedures allow any member of the community
> to appeal a decision by the IESG to start evaluating/voting on a
> document on the grounds that too many changes were made during
> (or after) Last Call to permit a fair judgment of community
> consensus.  Similarly, a Protocol or Document action notice
> could be appealed on the grounds that too many substantive
> changes have occurred (either since the IESG evaluation or the
> Last Call started) to be consisting with fairness in assessing
> community consensus.  I do not believe either of those appeal
> opportunities has been abused since RFC 2026 was published in
> 1996.  It might improve both fairness and opportunities for
> quick airing and resolution of issues if the IESG created
> explicit (to the community, not just IESG members and careful
> watchers) benchmarks for "starting IESG evaluation" and "about
> to go to the RFC Editor" with short windows of opportunity for
> "maybe not quite yet" comments at those stages (perhaps quick
> notes to the Last Call list).  They would not eliminate the
> possibility of appeals and could presumably be overlapped with
> other activities (e.g., IESG agenda setting) but might provide a
> more reliable and less time-consuming way to identify any issues
> that had arisen after the WG's publication request.
> 
> Those suggestions are independent of each other -- if some do
> not resonate, please consider the other(s).  FWIW, I believe
> that, in addition to improving fairness and the quality of the
> assessed IETF consensus and the documents themselves, they would
> also make the IESG review process more efficient and reduce
> burdens on IESG members.
> 
> thanks,
>    john
>