Re: draft-housley-two-maturity-levels

[John C Klensin <[e-mail]>]

--On Tuesday, August 02, 2011 20:23 -0400 "Joel M. Halpern"
<[e-mail]> wrote:

> With mild apologies, I have retained John's text below
> because, even though I come to a different conclusion, I
> thought it important to retain for now.  If folks choose to
> follow up on this, significant trimming is recommended.

Trimmed :-)

> John, as far as I can tell there are three problems which
> various folks have wanted this document or its predecessor to
> address:
> 1) That the bar for PS is too high
> 2) That not enough documents are advancing up the standards
> track
> 3) That what we do and what we say we do do not match.

I agree with your summary so far.

> Clearly, these problems are related.

That is less clear to me.  While I think that (1) is at least
part of the cause of (2), both of the following seem almost
equally plausible:

	(i) At the time the current model was assembled, IETF
	participation was much more heavily weighted toward
	researchers and others who had the flexibility to be
	able to concentrate on getting things right with
	multiple iterations and gradual advancement.  As the
	Internet has evolved toward an environment in which
	short-term product needs dominate, a larger percentage
	of IETF participants have needs to get a reasonable spec
	agreed on so it can be implemented and deployed, but
	little need to refine and revise unless flaws in the
	specs cause serious and user- or operator- visible
	interoperability problems.
	
	(ii) At the time the current model was assembled, IETF
	specifications tended to be for relatively simple
	protocols with either few options or orthogonal ones
	without complex interactions.  Today, we are seeing far
	more specs with optional variations, options that
	interact in complex ways, profiles for different
	purposes, etc.  We have whole working groups whose
	purpose it is to sort out the operational implications
	of protocols developed in other WGs (and sometimes still
	under development.   In that environment, having
	maturity categories that require an entire complex
	protocol, regardless of profiles or sets of options, to
	be classified into a small range of mutually-exclusive
	sets just makes no sense almost independent of how many
	categories we have.   Whether we can remove obstacles or
	not, we don't have the right incentives in place to get
	people to do things that make no sense to them (or to
	their sponsors).   Indeed, _increasing_ the number of
	categories might help: for example, perhaps we need a
	"base spec and 'mandatory to implement' pieces are fine,
	but we don't know about some of the option sets"
	category.

If some combination of those hypotheses is actually the driving
force behind few specs getting past Proposed, then nothing we do
about getting things into Proposed more easily will make any
difference.  Nor is fussing with the number of maturity levels
likely to have any useful effect.

In addition, (3) appears to me almost completely unconnected
with the other two.  Sure, we don't use annual review and never
have.  And it is definitely unattractive to have rules around
that we don't follow.  I would be a lot happier if we removed or
clarified _every_ rule in our specs that we don't follow (or if
we followed more of them).   But I haven't heard even a claim
that removing a rule that we haven't followed from 2026 would
make it easier to get documents approved as Proposed or that it
would increase, even slightly, the number of documents that are
advanced.

> We could try to adopt a change that would address all three.
> I dobut we would accomplish anything.

ok

> As far as I can tell, this document sensible tries to address
> one of these, namely item 3.  It tries to align what we
> document with what we do.  In order to do so, it also makes a
> change which may help item 2. It may not.

But there is no connection at all.  If we wanted to address item
3, then all it would take is a _really_ short RFC that updated
2026 and said "that provision has never been used and is now
eliminated".  As I indicated in my long note, I can't believe
there would be any controversy about such a document other than
complaints that we were wasting time that could be better spent
in other ways.

Might it help with item 2?  While I can't _prove_ it would not,
I don't recall seeing a single argument, either in the document
drafts or on the list, that justifies even a strong hypothesis
that the existence of a rule that we have never followed has
discouraged even a single specification.   If one thought there
was an interaction of that sort, it would make far more sense
logically to try applying the rule to see what effect that would
actually have.  I'm not suggesting that, if only because I
wonder whether moving RFC 3261, RFC 2616, or RFC 2460 to
Historic on the grounds of non-advancement would make us look
the most ridiculous.

> Now, you can argue that it is taking up energy that should go
> to item 1.   That is not unreasonable.  Except that I consider
> all the proposals I have seen for item 1 to be failures.  They
> fail in different ways, but they all have appeared to me to be
> bad ideas. 

Well, that is interesting. I'd be delighted to discuss some of
those proposals with you, but I don't believe that the community
has discussed any proposals that are intended to make
improvements on item 1 in at least the last half-dozen years.
Some attempts to have such discussions have been ruled out of
order on the grounds that they address a different topic than
draft-housley-two-maturity-levels.  Ones that merely would have
required the IESG to conduct a small-scale experiment have been
ignored.  Suggestions about others have been dealt with by
fairly clear statements that the IESG would not consider them as
individual submissions and that a WG would never be approved.

That may made all of them failures, but, if it does, there are,
IMO, some much more fundamental problems with our processes
than, e.g., whether the annual review rule is being used.

But, more important, I'm not making the argument that this is
"taking up energy that should go to item 1" (even though I
believe that to be true).  I think this document should stand
(or not) on its own content and what it proposes.   But it only
solves item 3 while making much more significant and noticeable
changes --look at its title for starters-- in the area of item
2... with no real justification for believing that it will have
any positive effect on the issue you identify as item 2
whatsoever.

I also believe that, if it is approved, it will be used as a
further excuse to not take up any proposals that might actually
solve blocking problems.  But I consider that an almost-separate
issue as well because, if this would really be effective against
_either_ issue 1 or 2, that would be worth the price.

> As such, we can either do nothing, or take this step that in
> my personal opinion addresses item 3, might turn out to help
> item 2, and does not hurt item 1.

See above.  I think that making a very significant change in the
standards process -- changing the number of maturity levels and
criteria for them certainly is such a change-- requires more
justification for believing it might be effective than "might
turn out to help".  The latter looks too much like
decision-making by throwing pasta at the wall... and using only
a single strand of pasta.

In addition, while I think it unlikely, there is a rational
argument (outlined in my earlier note and notes by others) for
believing that lowering the number of maturity levels and trying
to accomplish maturity level changes without new documents will
increase the pressure to get Proposed Standards just right.
Pressure to get Proposed Standards just right appears to be one
of the primary causes of the high threshold for those documents.
There is actually a stronger argument that this might make item
1 worse than there is that it will make item 2 better.


> I believe I understand your point below that without
> understanding how we ought to address problem 1, it is hard to
> be confident we are not making it worse rather than better.

Thanks,
   john