Post-Last-Call versions of documents and change logs: suggestion to the IESG

[John C Klensin <[e-mail]>]

IESG,

While neither issue is new, I've been troubled in the last few
weeks by a pair of related issues.  I want to identify them (as
briefly as I can) and, while it won't solve them, make some
small practical suggestions.  I am writing about WG documents
but, especially with various interpretations of RFC 8789,
AD-sponsored individual ones may be even more subject to
problems.

In the last few steps of our document approval process, an IETF
Last Call is announced and announced about a specific version of
a document.  If there are useful (typically as judged by the
authors, not, e.g., the WG that is responsible for the document)
comments during Last Call, the document is revised during and/or
immediately after that period, sometimes more than once.  After
the Last Call period ends, the IESG starts its formal review, in
theory making a determination of community consensus based on
the Last Call comments.    In practice, the ADs often find
issues not identified during Last Call and their comments often
lead to further document changes before a draft (possibly with
notes) goes off to the RFC Editor.  Over the years, some of the
issues caught in IESG review have been very important so the
present model, as practiced, is is probably nearly as good as we
can do.  However, it raises two very fundamental questions about
IETF consensus (rough or otherwise) about the finished work:

(i) Is the document on which the IESG review starts
substantively the same as the one on which the Last Call was
started?  Would any of the changes have elicited addition
substantive community comments had the been in the document when
the Last Call started?  Presumably both questions are answered
by the responsible AD in allowing the document to move into IESG
evaluation, but that means one person, with an investment in the
document and the WG, is making a potentially significant
decision on behalf of the community.

(ii)  When changes are made to the document at IESG request (or
even late discovery of issues by the authors), does the document
forwarded to the RFC Editor still represent community consensus?
Is that question even explicitly asked in all cases?

These two stages of the process are, at least in principle, even
more important than the AUTH48 one that has drawn recent
attention.   I don't think there is a "solution" to either, but
have some suggestions to mitigate the possible risks of a
document being published that does not represent community
consensus.  AFAICT, none of them require revisions to normative
process documents or rules.

(1) While it has never been a requirement (and, IMO, should not
be), many WGs and authors have included "Change Logs" that
summarize differences among versions of an I-D.  If new versions
of an I-D are produced after the Last Call starts, the IESG
should insist on such summaries if the modified versions are to
be considered without restarting the Last Call using those
versions.  That would allow community members who have already
reviewed and commented on a previous version, are working on
comments, and even those who have decided they do not need to
post Last Call comments based on a earlier version to swiftly
determine whether they need to reexamine the I-D or particular
sections of it.  While an irresponsible or malicious editor
could game that requirement in the hope of slipping something
through, that risk is better dealt with by choices of editors
(or firing them if needed) than by eliminating the benefits of
responsible behavior.

(2) While there are many advantages of making changes to
documents by Github pull requests, they do not make it easy for
someone concerned about aspects of a document but without the
time to get fully drawn into the work (IMO, exactly the people
IETF Last call is intended to reach because those who are more
involved presumably would have been part of the WG) to
understand all changes in context.  Probably there should be a
clear requirement that significant changes (e.g., replacement or
additions of whole sections or appendices) be accompanied by a
new draft so that it is possible to do a diff.  Even more
important, if _any_ changes were made during Last Call (or
during AD review before IESG review starts), a new version
should be posted so that the community can be aware of exactly
what the IESG is evaluating.  While I am less concerned about
it, it would be good for the IESG to consider whether a new
draft should be required along with a Protocol Action or
Document Action notice rather than reflecting any but the most
trivial of correction in IESG notes to the RPC.

(3) One way to mitigate some of the perceived problems would be
for the IESG to be much more ready to push documents back to the
originating WGs when significant issues are raised or changes
made, requiring at least a brief WG Last Call.  That would
assure that there is at least still WG consensus around the
post-Last Call version of a document the WG presumably produced.
It would obviously add an extra week or two to the process when
things go smoothly and so should be used with discretion.  But
it would, in some (many?) cases be faster and more consistent
with fairness than an extended discussion or negotiation between
authors and IESG members.  It would also be consistent with a
view that, if significant changes are needed to a document after
Last Call starts, they are often indicative of insufficient
diligence on the part of the WG to be sure that necessary
substantive issues have been reviewed and considered before
publication is requested.

(4) AFAICT, current procedures allow any member of the community
to appeal a decision by the IESG to start evaluating/voting on a
document on the grounds that too many changes were made during
(or after) Last Call to permit a fair judgment of community
consensus.  Similarly, a Protocol or Document action notice
could be appealed on the grounds that too many substantive
changes have occurred (either since the IESG evaluation or the
Last Call started) to be consisting with fairness in assessing
community consensus.  I do not believe either of those appeal
opportunities has been abused since RFC 2026 was published in
1996.  It might improve both fairness and opportunities for
quick airing and resolution of issues if the IESG created
explicit (to the community, not just IESG members and careful
watchers) benchmarks for "starting IESG evaluation" and "about
to go to the RFC Editor" with short windows of opportunity for
"maybe not quite yet" comments at those stages (perhaps quick
notes to the Last Call list).  They would not eliminate the
possibility of appeals and could presumably be overlapped with
other activities (e.g., IESG agenda setting) but might provide a
more reliable and less time-consuming way to identify any issues
that had arisen after the WG's publication request.

Those suggestions are independent of each other -- if some do
not resonate, please consider the other(s).  FWIW, I believe
that, in addition to improving fairness and the quality of the
assessed IETF consensus and the documents themselves, they would
also make the IESG review process more efficient and reduce
burdens on IESG members.

thanks,
  john