Re: Things that used to be clear (was Re: Evolving Documents (nee "Living Documents") side meeting at IETF105.)

[John C Klensin <[e-mail]>]

(Long message -- summary a few paragraphs down)

Hi.  

Following this thread and the ones it has spawned have proven
nearly impossible for me, especially in the run up to IETF 105
and the need to get many other things done if I'm going to
attend the meeting.  I have no way to know whether others (and
how many) have suffered from the same problem or who see these
threads (and the distantly-related RSE ones) as DoS attacks on
the IETF list.  I have certainly failed to read many messages
and apologize if I've missed anything significant.  

However, I started writing a note over a week ago that I think
is relevant to much of the discussion and whose content doesn't
seem to have been covered by other messages, possibly because
my history and perspective are a bit unusual among IETF
participants.  The issues may be important because they
ultimately relate, not just to the RFC Editor Function and
documents that are more authoritative and final than I-Ds but
less than RFCs, but to fundamental questions about how we got
here, how that relates to why the Internet took off and other
alternatives didn't, and where we are headed.

     ----

TL;DR summary: (1) There is a conflict of roles inherent in the
IETF's history, how it has operated until fairly recently, and
how its ways of doing things seem to be evolving and
transitioning.  (2) Part of our traditional narrative
(partially from long before the IETF) about why the Internet
protocols succeeded and various of the alternatives, notably
the OSI suite, did not is as much about failures on their part
as about inherent brilliance on ours.  (3) Several of the
things that have been proposed are normal in traditional
standards bodies and we should be careful what we wish for.
And, I suppose, (4) been there, tried that, didn't like it.  

What follows could probably be expressed much better and far
more succinctly by some of our present and former colleagues
who are or were inclined toward acidity of the tongue and, in
some cases, even name-calling.  I hope what follows is much
more respectful and appropriate, but that is part of what makes
it so long. 

    ------

Some of what follows is historical review. My apologies to
those who know most or all of it already, but, even for them,
the context may be a bit different than the way we've
traditionally talked about these things.

The IETF and its predecessors started out as engineering design
groups.  They made the decision very early on that an archive
of what others might have called "design notes" or "informal
working papers" would be helpful, an archive that became the
RFC Series (see RFCs 1 and 3 and draft-flanagan-fiftyyears for
details, perspective, and references).  Even before the IETF
was established, Jon Postel became RFC Editor and the shift
from shared (and archived) design notes toward a curated
collection that had to conform to Jon's (and later Jon's and
Joyce Reynold's) editorial and technical quality norms rather
than just being documents that anyone in the community could
have published without prior review by anyone else.  Calling
some of the RFCs that were produced "Standards" came later and
thinking of and describing ourselves as a Standards Developer
came later still.  If the details and exact chronology are
needed, I hope someone else will fill them in because I wasn't
paying very much attention when some of the key decisions were
made.

There is an intrinsic tension between the needs --and
appropriate and optimal document identification systems-- for a
standards development body relative to an engineering
development and design one.  It is not clear that it is
possible to optimize both at the same time, nor that the IETF
has ever gotten the balance right.  As has been pointed out in
this and the various RFC Editor-related threads, there have
been a number of proposals over the years to make things better
by either tuning our documents or tuning the standards process.
The first major change after Jon Postel took control of the
series was the introduction of Internet-Drafts which, AFAICT,
were intended to serve nearly the same "quick availability of
ideas for consideration by others" function that the RFC Series
was intended to serve before Jon began to insist on quality and
stylistic norms for RFC publication.  

As the IETF gradually became more of a standards developer, it
not only decided that it should publish its work in the RFC
Series and that some RFCs should be referred to as standards
but established conditions and terminology for those standards.
Those efforts culminated in the description of the standards
track in RFC 2026 (and subsequent modifications that don't have
much to do with this discussion except as noted below).  It was
probably wishful thinking (and, in retrospect, it definitely
was) but the intended criteria for Proposed Standard was
community consensus that a specification was adequately
complete and clear enough technically and editorially that
consistent implementations were plausible, not that it was a
good idea or even nearly ready for large-scale deployment.  It
was a finished proposal, not a standard.  Such a proposal is
several steps above a working paper but still a step below
anything the IETF was recommending deploying or really treating
as what anyone else would call a "standard".

In the last decade or two, only two of the proposed
improvements got any significant traction in the community (and
got a chance to get that traction).  One was the series of
changes to, and specific definitions for, the organization of
the RFC Editor Function that followed Jon's untimely passing
and Bob Braden's making it clear that he did not expect to be
the lead on the RFC Editor Function forever.  The other was the
standards process change in RFC 6410, a change that was
intended to improve on "the Internet runs on Proposed
Standards".  As I think I suggested in an earlier note, it,
empirically, didn't cause great change.  As least some of those
of us who believed it was solving the wrong problem think our
opinions were validated.  I don't think we've ever heard an
evaluation from those who thought it would move significantly
more specifications off Proposed.

The whole idea of a multistage standards process was, in this
context, an attempt to reconcile the engineering desire to be
able to get a stable specification that could be implemented,
tested, and shown to interoperate with more traditional
normative standards, including those focused on conformity.  In
other standards bodies, not only are documents either
"standards" or "drafts", but the drafts are often treated as
confidential within the development group.  In other words,
they are not available to the public while they are actively
being worked on.  They are definitely not archived and easily
found, forever, by anyone who is interested.  There are a few
exceptions.  For example, IEEE has "trial-use" specifications
and some other categories that are different from traditional
standards
(https://standards.ieee.org/about/policies/opman/sect1.html).
Long ago, when I was a liaison member of the IEEE Standards
Board, my impression was that they didn't work well.  In
particular, I heard some of the same complaints we hear in the
IETF, e.g., some companies marketed products based on them as
stable and standards-conforming and implementers whined when
they implemented a trial-use document and, when the final
document came out, something got changed in an incompatible
way.  I have no idea whether our three-stage model influenced
the IEEE, if theirs of different classifications influenced us,
or if they were independent inventions.  

Would we have been better off it we had called untested
specifications "Trial Use" rather than "Proposed Standard"?  My
guess is probably yes, but not much.  I think our experience
--both inside and outside the IETF-- is that, if people want to
misunderstand or misrepresent that status of a document, they
will do so regardless of what it is called.  That same
experience suggests that actual misunderstandings, especially
by people who actually look at the documents, glance at status
statements, etc., are fairly rare.

On a different dimension of the problem, ISO/IEC JTC1 has, or
had (I vaguely recall being told they had gotten rid of them),
a variety of technical report that allowed publishing
specifications that were either significantly pre-standard or
that had not gotten and were unlike to get consensus outside
the developing WG (and whose level of consensus within the
developing WG could be what we'd call quite rough) -- the
latter and "pre-standard" are, of course, not necessarily
different-- (see
https://www.iso.org/obp/ui/#iso:std:iso-iec:tr:10034:ed-1:v1:en
for more information although there are probably better
references).  Those "type 2 technical reports" came with
multiple warnings and disclaimers about how one couldn't claim
conformance to them and that they didn't constitute
recommendations.  They were affectionately known as
"substandards" internally.  Our nearest equivalent is an I-D,
but, while our boilerplate claims that I-Ds expire, we do too
much to ensure that they are permanently available for that to
be credible, especially if someone is inclined to mislead
others about the status of a document.   Have marketers claimed
from time to time that Type 2 Technical Reports are standards?
Yep, even though ISO and JTC1 have better tools to push back
against that game than we do.  Of course, their not making
working drafts public (and protecting that status with strong
membership agreements) at least helps to keep the status of
those drafts from being misrepresented in marketing although
not from premature implementations.

Those other standards bodies have also been clear about the
boundaries of their standards.  Standards numbers are permanent
and specifications are not considered archival.  When a
standard is extended, there are mechanisms for establishing
"parts" or other components, documents are referred to by date
as well as number, and, not incidentally, there are formal
review cycles so that old and obsolete (sic) documents don't
hang around forever as standards or require cruft-elimination
efforts (see RFC 4450).  STD numbers were supposed to solve
that problem for us, but they were tied to full standards (of
which we don't have enough to make them useful).  We also
discovered that the way we develop documents (that engineering
history again) makes deciding whether a spec is, or is not,
part of a given standard... well, sometimes not obvious.  There
have been proposals to fix the numbering and document
relationship problem (most notably in the NEWTRK WG), none of
which got the IESG to agree to process them, even when the
proposals came out of a WGs and had WG consensus.

The notion that, e.g., ITU-T has better-quality documents that
we do is another one of those choices.  ITU-T (and ISO,
including ISO/IEC JTC1) believe that one should do final public
review and ballots only on final documents, "final" including
editorial work.  If the public review and ballot process turn
up problems, even issues requiring clarification, the normal
procedure involves editing, a new draft, and a new ballot.
Documents go into public review with text in need of
clarification less often than it does with us because theirs
have already been gone over carefully by professional technical
editors who have the authority to either rewrite or to bounce
things back to the WG (or SG or SC) before public review is
opened (in theory, the RFC Editor has the ability to bounce a
document that they can't figure out how to edit, but that has
very rarely been done in recent years (I don't have statistics,
but I'd guess just about never).  In any argument between the
editors and, e.g., a WG (or SG --their closest approximation to
our Areas) about whether something is clear and unambiguously
stated, their editors always win while ours are required to
defer to authors, the IESG, etc.  

Instead, we approve documents in principle, assume editorial
issues will be sorted out later (nit-picking aside) and give
the RFC Editor staff, the IESG, authors, and sometimes WG
chairs and a few others tremendous discretion about what
changes can be made post-approval.  Our way of doing things is
probably faster, doesn't leave WGs sitting around for an
extended period while their documents are edited or tie them up
in editorial discussions, and, most important, gets the
technical content of specifications that have gotten IETF
consensus out there, rather than waiting until after the
editing is done before determining consensus.  Contrary to some
comments about cost, the ITU/ISO way probably isn't that much
more expensive in the normal case.  The issue is mostly about
whether to edit early or edit late with the understanding that
editing between when the WG thinks it is finished and when a
Public Review (them) or Last Call (us) occurs stretches out
that time, often considerably.  On the other hand, if a
document were edited before Last Call and significant changes
made during the LC and IESG review process, it would need to be
edited again -- much of that would add costs.

Coming back to the issue of whether document status may be
misunderstood, we also know, because we have rather intense
discussions about it every few years, that people confuse
non-standard "IETF documents" and non-standard RFCs with
standards.  As suggested above, in most of the cases I know
about, that confusion is not accidental even though there may
be innocent victims.  Instead, we see organizations promoting
their products by claiming that they are based on "IETF
specifications" (i.e., I-Ds or Informational or Experimental
RFCs regardless of Stream) and implying those are standards.
Or they may claim "published by the IETF" and close to
standardization (i.e., I-Ds) or even imply that those are
standards.  We are not going to prevent any of that by changes
in terminology or boilerplate: most of the liars know they are
lying.  

If we are concerned about those problems, probably the last
thing we need to do is to create a new category that provides a
declaration by a WG -- whether there is IETF consensus about it
or not -- that something is finished and stable enough to be
used in deployed products.  The intent may be clear and
positive, but the opportunity for unscrupulous marketers is
just too great.  So are the incentives to just not bother going
through an IETF Last Call and approval process that might
change things and destabilize the spec.  

In particular, suppose we have a WG (or set of authors or Area)
that doesn't see any incentive to move a specification to
Internet Standard from Proposed even though it has been around
for several years and has been widely deployed and demonstrated
to interoperate.  Shouldn't that WG reasonably be considered a
candidate for declaring a document stable (and at least mostly
done) and then not trying to move it to Proposed Standard?
After all, most WGs are convinced that they understand the
implications of their specs and that additional work is not
needed or they would not push for IETF Last Call.  Why should
they invite the things that go with IETF LC including the
procedural delays, hassle, nit-picking, second-guessing, and
potential to be forced to make the specification worse to
accommodate various ADs that have their own ideas about what it
should say or how it should say it?

Again, we know from other standards bodies how to mitigate (but
not eliminate) most of those problems with "trial",
"preliminary but stable", pre-standard technical reports, and
similar documents.  The main tool is to attach a rigid
expiration date after which, if the the document has not been
turned into a standard, it is automatically and clearly
deprecated and authoritative copies are made hard to find.  But
we've tried that.  RFC 2026 provides for automatic forced
review for Standards Track documents that have been in a
maturity level for too long, but we never applied it as a
forcing function and eventually just deprecated it.  Our
original definition of a Proposed Standard differs from a
WG-declared stable specification only in that IETF rough
consensus was required (and it was allowed to be a lot rougher
when we thought that people would actually accept Proposed
Standards as what we said they meant).  

More generally, one of the things that makes the IETF credible
is the notion of cross-area review.  We need to _expect_ things
to change as the result of IETF Last Call comments.  Certainly,
if we can get earlier reviews on specific topics and thereby
improve the documents, that would be A Good Thing.  But Last
Call is intended to turn up "you didn't think about the
consequences of that" answers, especially those that come as
surprises.  Recent discussions of layer violations
notwithstanding, we rarely publish a standards track RFC that
seriously messes up things in a different area.  At least some
of us believe that is due to precisely those reviews (whether
by individual participants, area review teams, or ADs) catching
issues that a WG (especially an excessively homogeneous one)
might not.  Allowing such a WG to declare a document "stable"
or "ready for implementation" without that Last Call review, is
an invitation for abuse, misrepresentation, and avoidance of
final review, even if none of the WG participants are among the
bad actors.

  best,
   john