Moving forward on IETF problems

[Bruce Lilly <[e-mail]>]

This has been an interesting discussion, but somewhat one-sided.
Authors, editors, and WG Chairs and participants tend to believe
that they produce perfect output, but that is demonstrably not
always true.  Although the IESG supposedly provides a quality
control, it's clearly not working.  Rather than vague assertions,
I'll mention a couple of specifics: RFC 4021 and draft-fax-esmtp-conneg.

RFC 4021 has been published as Standards Track. In accordance with
RFC 2026, it contains a section referencing RFC 2119 and the latter's
keywords which indicate levels of requirements.  However, none of
those keywords appear anywhere in the document but the section stating
that 2119 in fact defines such keywords!  Moreover, the document
does not impose any protocol or format requirements and does not lend
itself to any implementations (its purpose is to document items being
included in an IANA registry); the Standards Track is the wrong
category for such a document -- there is no way that it can advance
to Draft Standard.  This is a spectacular and surprising instance of
complete failure of the authors, the IESG, and the RFC Editor to
control quality.  It is surprising and spectacular because the
document structure is so simple and its purpose evident.  There is
no way that the IESG and RFC Editor should have categorized that RFC
as Standards Track.

There are many issues with the draft mentioned above; despite having
ABNF expertise among the authors, there are at least a half-dozen
specific problems with the ABNF.  The document lacks discussion of
interaction with directly-related protocols (ESMTP submission, e.g.),
has no BCP 90 registration forms for message header fields defined,
no discussion of internationalization issues, and an inadequate
IANA considerations section.  The draft has been through at least
one Last Call (version -09 and possibly also version -02), and has
been approved as a Proposed Standard (-13) despite the known technical
omissions, the vagueness which precludes the specification from
being well-understood, etc.  This is another failure of quality
control; the document does not meet the minimal requirements for
Proposed Standard (RFC 2026 section 4.1.1).

There are clear requirements for the Standards Track; apparently
the authors, WGs, the IESG and the RFC Editor aren't taking the
necessary time to review documents for compliance with those
requirements.  I have separately (NEWTRK mailing list) suggested
that those requirements should be distilled into a checklist, and
have further suggested that some tolls might be capable of
highlighting specific problems such as the RFC 4021/2119 issue.

Clearly, I'm bemoaning the lack of quality of recent work; others
have complained about timeliness.  Still others have mentioned
relevance and comprehensibility.  There have been some allusions
to cost (in the form of increased manpower for review).  There is
a well-known engineering issue:

   o Quality
   o Timeliness
   o Cost
   Pick any two.

Part of the problem with time/quality tradeoff is that many drafts
are in such a poor editorial state that it impedes review.  One
possible way to expedite the technical review would be to ensure
that documents are edited for readability and comprehensibility
prior to IESG review and Last Call.  That may involve cost (somebody
has to provide editorial advice).  Or the IESG could simply push
back documents of poor editorial quality (ideally with a list of
resources -- currently scattered among RFCs, presentations, semi-
official documents (ID-guidelines, ID-Checklist, ID-nits), etc.).

RFC 3774 has been mentioned in this discussion; that document
and RFC 3844 discuss some problems and potential solutions, but
do not address the generally poor quality of documents entering
the review stage.  Some of the issues identified have been partially
addressed (e.g. RFC 3935 provides the mission statement mentioned
as lacking in section 2.1 of RFC 3774 -- but the IETF web site
makes no mention of the IETF mission and has no direct link to the
mission statement).  3774 also bemoans a perceived lack of assessment
based on practical experimentation, ignoring the fact that most
RFC categories specifically provide for such practical experimentation
(i.e. the Standards Track plus Experimental (and Historic for failed
experiments)).  3774 also makes the extraordinary claim that the
quality bar has been raised.

As mentioned above, a part of the quality problem can be attributed
to lack of organization of relevant material in a single place. The
NEWTRK "ISD" effort might be able to help -- but as noted above, many
of the relevant information is in documents that are not RFCs, and
which may have no clear official status, and moreover, which may
change frequently.  However, there is a danger that the benefit of
consolidating pointers to related information may be diluted by
incorporation of unrelated agendas (specific document format
issues), and the danger of causing a further deterioration in
quality (by weakening the iterative improvement process embodied
in progression along the Standards Track).  And it's not at all clear
that making information more readily accessible will improve the
quality of documents; the examples of quality control failures
given above cannot be attributed to ignorance of relevant requirements.

Some relevant information could be more widely disseminated with clear
benefit; for example, some boilerplate pointing to the RFC Editor's
errata page could be included in every published RFC, informing
readers of the mechanism for documenting relevant updates.

One problem is that the IESG routinely sabotages development along
the Standards Track by disbanding WGs as soon as a PS is produced,
leaving nobody to do the work necessary for advancement to Draft.
Charters should probably explicitly provide for WG activity leading
at least to Draft Standard (or to Historic if the necessary two
independent implementations fail to develop within a reasonable time).

_______________________________________________
Ietf mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/ietf