Review of: Characterization of Proposed Standards
Dave Crocker <dhc@dcrocker.net> Thu, 24 October 2013 13:29 UTC
Return-Path: <dhc@dcrocker.net>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 575D411E81A1 for <ietf@ietfa.amsl.com>; Thu, 24 Oct 2013 06:29:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Level:
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TG0yawXGjOcJ for <ietf@ietfa.amsl.com>; Thu, 24 Oct 2013 06:29:28 -0700 (PDT)
Received: from sbh17.songbird.com (sbh17.songbird.com [72.52.113.17]) by ietfa.amsl.com (Postfix) with ESMTP id 83CB811E829B for <ietf@ietf.org>; Thu, 24 Oct 2013 06:29:25 -0700 (PDT)
Received: from [10.3.89.42] (IP-173-231-109-130.static.fibrenoire.ca [173.231.109.130] (may be forged)) (authenticated bits=0) by sbh17.songbird.com (8.13.8/8.13.8) with ESMTP id r9ODTA6l017680 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 24 Oct 2013 06:29:14 -0700
Message-ID: <5269209F.3060706@dcrocker.net>
Date: Thu, 24 Oct 2013 09:29:03 -0400
From: Dave Crocker <dhc@dcrocker.net>
Organization: Brandenburg InternetWorking
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1
MIME-Version: 1.0
To: draft-kolkman-proposed-standards-clarified.all@tools.ietf.org
Subject: Review of: Characterization of Proposed Standards
Content-Type: text/plain; charset="ISO-8859-1"; format="flowed"
Content-Transfer-Encoding: 7bit
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (sbh17.songbird.com [72.52.113.66]); Thu, 24 Oct 2013 06:29:15 -0700 (PDT)
Cc: IETF Discussion <ietf@ietf.org>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: dcrocker@bbiw.net
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ietf>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 24 Oct 2013 13:29:33 -0000
Review of: Characterization of Proposed Standards I-D: draft-kolkman-proposed-standards-clarified-04 Reviewed by: D. Crocker Date: 24 October 2013 Summary: This draft offers revised normative text for the criteria used to qualify IETF drafts as entry-level Proposed Standards status. The draft's background discussion makes a number of assertions of fact without substantiation, at least two of which are probably not correct. The revised normative text includes two criteria that also are likely inaccurate. The draft is not ready for approval. Detail: > Abstract > > RFC 2026 describes the review performed by the IESG on IETF Proposed > Standard RFCs and states the maturity level of those documents. > This document clarifies those descriptions and updates RFC 2026 by > providing a new characterization of Proposed Standards. "those" descriptions? As the Introduction confirms, the draft covers only one of them. > 2. IETF Review of Proposed Standards > > > The entry-level maturity for the standards track is "Proposed > Standard". A specific action by the IESG is required to move a > specification onto the standards track at the "Proposed Standard" > level. Initially it was assumed that most IETF technical > specifications would progress through a series of maturity stages > starting with Proposed Standard, then progressing to Draft Standard > then, finally, to Internet Standard (see RFC 2026 section 6). Over > time, for a number of reasons, this progression became less common. It was /always/ uncommon. There was never a time when progression through the full sequence typically happened. > In response, the IETF strengthened its review of Proposed Standards, > basically It was not 'in response'. There was never a process of discussing the issue and changing a policy formally and with clear IETF rough consensus. The higher bar for Proposed happened organically and mostly through many independent decisions of those running the standards process over the years. Even better, it was always controversial, when discussed in public. Or rather, the question of how high the bar should be was always controversial and was never subject to an explicit rough consensus process. > operating as if the Proposed Standard was the last chance for the > IETF to ensure the quality of the technology and the clarity of the > Standard Track document. The result was that IETF Proposed > Standards approved over the last decade or more have had extensive > reviews. Many have. Many have not. It has depended upon the diligence chosen by everyone working on the draft or doing reviews. I'm pressing this point because the language being used for this draft is leaning towards mixing formal process measures to raise the /effort/ needed to get approved, with actual quality assurance /results/. In fact we often publish drafts that have actually have very few eyes doing deep and thoughtful review and we often publish drafts that don't work very well. That is, the much higher formal bar still allows quite a bit of cruft to jump over... > Because of this change in review assumptions, IETF Proposed > Standards should be considered to be at least as mature as final > standards from other standards development organizations. An assertion like this requires some discussion detail that compares the IETF process to specific, other SDO processes, both formally and effectively. I'm not disagreeing with the conclusion, but with its being made rather casually and firmly, but without substantiation. (Part of the real challenge in doing the comparison is looking for alternative forms of quality assurance that other SDOs might do, other than the explicit reviews the IETF does.) By the way, such a comparison discussion will probably serve to greatly strengthen the ability to explain/argue for the comparative maturity of Proposed... > The IETF review is possibly more extensive than that done in most > other SDOs owing to the cross-area technical review performed by the > IETF, exemplified by technical review by the full IESG at the last > stage of specification development. That position is further > strengthened by the common presence of interoperable running code and > implementation before publication as a Proposed Standard. There's actually a reasonable possibility that we do /less/ pre-Proposed interoperable implementation now than we did 20 years ago. Consequently an assertion that implementation is 'common' needs some objective substantiation in comparison to the time we defined the standards labels. > 3. Characterization of Specification > > > Section 3.1 of this document replaces RFC 2026 Section 4.1.1. > Section 3.2 is a verbatim copy of the characterization of Internet > Standards from RFC 2026 Section 4.1.3 and is provided for convenient > reference. > > 3.1. Characterization of IETF Proposed Standard Specifications > > > The entry-level maturity for the standards track is "Proposed > Standard". A specific action by the IESG is required to move a > specification onto the standards track at the "Proposed Standard" > level. > > A Proposed Standard specification is stable, has resolved known > design choices, is well-understood, has received significant > community review, and appears to enjoy enough community interest to > be considered valuable. The text has two major flaws that serve to assert a much higher practical maturity for the Proposed Standard documents we publish than is very often observably true: Well understood: For a specification with any interesting level of innovation or complexity, it is almost never 'well understood' at the time it is approved by the IETF. That requires significant implementation, deployment and use experience. The fact that many eyes might have done a static review or even that a couple of folk have written some code or even that those folk interoperated does not come close to qualifying the specification as "well understood". Enough community interest: We often publish specifications that actually have very little community support. This is evidenced by a) very low working group participation levels, and b) failure to successfully deploy and get used. > > Usually, neither implementation nor operational experience is > required for the designation of a specification as a Proposed > Standard. However, such experience is highly desirable, and will > usually represent a strong argument in favor of a Proposed Standard > designation. > > The IESG may require implementation and/or operational experience > prior to granting Proposed Standard status to a specification that > materially affects the core Internet protocols or that specifies > behavior that may have significant operational impact on the > Internet. > > A Proposed Standard will have no known technical omissions with > respect to the requirements placed upon it. Proposed Standards are > of such quality that implementations can be deployed in the > Internet. And yet we know that this often is not true. However, language like this is going to cause the IESG to naturally raise the bar even higher, for fear that a spec still is not good enough. The real goal of the language change, here, is to give assurances that the specification is "sufficiently" mature, in terms of concern for market deployment. The part that the document can't say, but has as an implicit point since it is motivating the draft, is that the maturity is at least as good as that produced by other SDOs. Perhaps: Proposed Standards are of such quality that they are ready for the usual market-based product development and deployment efforts into the Internet. > However, as with all technical specifications, Proposed Standards > may Remove "However". > be revised if problems are found or better solutions are identified, > when experiences with deploying implementations of such technologies > at scale is gathered. > 4. Further Considerations > > > While less mature specifications will usually be published as > Informational or Experimental RFCs, the IETF may, on occasion, On occasion? In fact it's quite common and possibly the norm. Let's not pretend otherwise. > publish a specification that still contains areas for improvement or > certain uncertainties about whether the best engineering choices are > made. In those cases that fact will be clearly and prominently > communicated in the document e.g. in the abstract, the > introduction, or a separate section or statement. d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net
- Review of: Characterization of Proposed Standards Dave Crocker
- Review of: Characterization of Proposed Standards Abdussalam Baryun
- RE: Review of: Characterization of Proposed Stand… Moriarty, Kathleen
- Re: Review of: Characterization of Proposed Stand… Dave Crocker
- Re: Review of: Characterization of Proposed Stand… Olaf Kolkman
- Re: Review of: Characterization of Proposed Stand… Olaf Kolkman
- Review of: Characterization of Proposed Standards Abdussalam Baryun
- Re: Review of: Characterization of Proposed Stand… Dave Crocker
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… SM
- Re: Review of: Characterization of Proposed Stand… Olaf Kolkman
- Re: Review of: Characterization of Proposed Stand… Olaf Kolkman
- Re: Review of: Characterization of Proposed Stand… SM
- RE: Review of: Characterization of Proposed Stand… Larry Masinter
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… Dave Crocker
- Re: Review of: Characterization of Proposed Stand… Olaf Kolkman
- Re: Review of: Characterization of Proposed Stand… Jari Arkko
- Re: Review of: Characterization of Proposed Stand… Jari Arkko
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… Dave Crocker
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… Dave Crocker
- Re: Review of: Characterization of Proposed Stand… Brian E Carpenter
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- Re: Review of: Characterization of Proposed Stand… Olaf Kolkman
- Re: Review of: Characterization of Proposed Stand… Barry Leiba
- The curse of Apple wasRe: Review of: Characteriza… t.p.