Re: draft-housley-two-maturity-levels

["Joel M. Halpern" <[e-mail]>]

With mild apologies, I have retained John's text below because, even 
though I come to a different conclusion, I thought it important to 
retain for now.  If folks choose to follow up on this, significant 
trimming is recommended.

John, as far as I can tell there are three problems which various folks 
have wanted this document or its predecessor to address:
1) That the bar for PS is too high
2) That not enough documents are advancing up the standards track
3) That what we do and what we say we do do not match.

Clearly, these problems are related.
We could try to adopt a change that would address all three.  I dobut we 
would accomplish anything.

As far as I can tell, this document sensible tries to address one of 
these, namely item 3.  It tries to align what we document with what we 
do.  In order to do so, it also makes a change which may help item 2. 
It may not.

Now, you can argue that it is taking up energy that should go to item 1. 
  That is not unreasonable.  Except that I consider all the proposals I 
have seen for item 1 to be failures.  They fail in different ways, but 
they all have appeared to me to be bad ideas.  (I think, based on 
conversations, some folks were supporting the previous version of this 
document in the mistaken believe that it did more to address that than 
it actually provided.)

As such, we can either do nothing, or take this step that in my personal 
opinion addresses item 3, might turn out to help item 2, and does not 
hurt item 1.
I believe I understand your point below that without understanding how 
we ought to address problem 1, it is hard to be confident we are not 
making it worse rather than better.

Yours,
Joel

On 8/2/2011 6:51 PM, John C Klensin wrote:
>
> On 7/30/11 11:05 AM, Joel M. Halpern wrote:
>> It seems to me that this does two things, both small but
>> useful. 1) It makes a minor change in the advancement
>> procedures so that they are more reasonable.  They may still
>> not be sufficiently reasonable to be used, but it improves
>> them, and thereby improves the odds.
>
> Actually, there is no evidence that this is an improvement.  I'd
> agree that it is a minor change, but there has been no serious
> analysis of whether it is an improvement or not.  To the extent
> to which approving this lowers the odds of considering and
> making changes that might actually have significant effects
> (e.g., really sorting out what maturity levels mean in a world
> of increasingly complex standards), it is harmful.  See long
> discussion below.
>
>> 2) It is coupled to an
>> intent to actually behave according to what the document
>> says.  Such an intent is obviously not feasible without some
>> change.
>
> Well, yes and no.  My sense of the discussion over the last year
> is that a significant fraction of the community believes that
> the critical path change in this area is an adjustment to the
> threshold for Proposed Standard.  That issue is addressed in
> draft-bradner-restore-proposed, which requires no change to RFC
> 2026 or other formal procedures at all.  It is not addressed in
> this document.
>
>>   It is useful to have our behavior and our documented
>> description of how we work match because the mismatch causes
>> confusion, at least for new participants, and sometimes even
>> for experienced participants.
>
> I agree with this.  But this document doesn't make nearly enough
> of a contribution in that area to justify approving it.  It
> addresses exactly one provision of our processes in which
> documentation and practice don't align, a provision about which
> there is no subtlety or confusion within the community at all
> (even though new participants may be confused).  If the issue is
> important, then let's make a serious effort to update the places
> where 2026, 2028, 2031, 2360, 3710, 4071, and probably several
> other documents have fallen at least somewhat out of line with
> reality.  If the particular issue of the annual review is really
> more important than any of those other issues, I assume that a
> stand-along document that changed it would rapidly achieve
> community consensus (albeit with some complaints about wasted
> time).  Certainly it should not be sufficient to justify
> approving this document... the change is almost irrelevant to
> it.
>
>> It might be the case that it will improve the advancement
>> percentage. It might not.  I can not imagine that it will
>> make that even worse.
>
> I can because the effects of changes like this are actually very
> hard to predict.  It increases the requirements for the second
> level, however slightly.  Since we have trouble getting things
> to the second level now, that increased requirement might reduce
> incentives to bring things off Proposed at a least as much as a
> theoretical "just one more level and then you are finished"
> change would increase those incentives.  By changing Proposed
> from "1/3 of the way" to "1/2 way or a bit more", it might
> remove a small incentive to take the additional step.
>
> In addition, the "publication without a new RFC" provision may
> actually further increase the de facto requirements for Proposed
> Standards by requiring that those documents be edited to a high
> standard of clear English and specification precision.  While,
> IMO, we have taken too little advantage of it, the current
> provisions of RFC 2026 permit publishing a Proposed Standard as
> soon as the WG understands it, leaving language cleanup (and
> refined translation from the writing styles of many people in
> the community whether native speakers of English or not) for
> Draft Standard versions.
>
> More important, for those of us who believe that a maturity
> system based on rigid named categories that apply to entire
> documents has outlived its usefulness as our specifications have
> become more complex, approval of this document is almost certain
> to cause consideration of such approaches to be postponed by
> some years as people complain that the changes made in this
> draft must have time to take effect and be evaluated.
>
>     ---
>
> A more extended analysis of other aspects of the situation with
> this document follows.  Those who don't like extended analyses
> might want to stop reading at some point.
>
>
>
> A very long time ago, a then-professor of mine observed that one
> of the most common sources of failures in engineering,
> especially computer engineering, was to look at a problem that
> seemed too large or too intractable, find some easily-changed
> and tractable part of the problem, do something with it (almost
> anything), and then claim that significant progress had been
> made on the original/ real problem because one had started on
> it.   In many cases, the approach actually makes the real
> problem harder: systems become more complex, applying remedies
> later turns out to be more complicated, and so on.  The narrow
> "solution" may also use up energy and creates expectations that
> make it harder than it would have been otherwise to take on the
> real work when the narrow job fails to solve any significant
> problems.
>
> An even more insidious version of the same approach is to
> identify a problem and claim it is serious enough that one needs
> to do _something_.  One then proceeds to do something that has
> nothing at all to do with the problem, just to demonstrate
> motion (or in the hope that setting a butterfly in motion in one
> part of the world will cause major change elsewhere).
>
> These approaches are very different from taking a large design
> issue and modularizing it into pieces that can be worked out
> separately but with clear and well-understood interfaces.  Doing
> that right requires that the overall design issue be understood
> well enough to define all of the modules and interfaces, not
> doing one piece, ignoring the poorly-understood rest, and hoping
> that things will sort themselves out.
>
> The IETF periodically falls into this trap.  We often see
> protocols that are designed to handle the easy issues adopted
> without consideration of the harder ones and edge cases, only to
> discover that those protocols represent a framework into which
> solutions for the other issues won't fit.  The largest symptoms
> of this used to show up only with Security issues -- protocols
> being designed without security provisions with the expectation
> that security  would somehow be patched in later... and despite
> repeated experiences that such patches are rarely completely
> satisfactory.    Now we see it spreading: I've seen "we knew
> about those issues but ignored them and now have to clean up the
> mess somehow" comments in two separate WGs in recent weeks and
> suspect that there are many more.  Many of the comments on this
> list try to justify approving this draft on the same basis.
>
> These "there is a problem so let's do something and hope it will
> help", "even though we can demonstrate no logic that would
> predict that the change being proposed will result in the
> desired effect", and "let's fix something to show that we can"
> approaches are _always_ justified on the basis of "baby steps"
> or "incremental efforts".   Those justifications would be quite
> reasonable if we actually had good evidence of the linkages.  In
> the absence of such demonstrated linkages, we are just thrashing
> around and wasting energy --energy that, under other
> circumstances, could actually be used to address the problems.
>
> So, while I applaud the efforts of the authors of this draft to
> remove controversial and largely extraneous material, it seems
> to me that the core issue remains the one that many people have
> commented on in much earlier versions: At present, we don't get
> many documents from Proposed Standard to Draft Standard, despite
> the requirements for Draft Standard being slightly lower than
> those set forth for Internet Standard in this document.  We have
> a possibly-serious problem with the norms for Proposed Standard
> being too high in practice --far beyond what RFC 2026 requires.
> It is noteworthy that draft-housley-two-maturity-levels no
> longer makes any claim to solve that problem.  On the other
> hand, there has been considerable discussion in the community,
> previously supported by the strongest advocates for the
> predecessors of the current draft, that the norms for getting to
> Proposed Standard _are_ the keystone problem.  I don't
> personally believe that -- I think things are more complicated--
> but I do see it as a critical element (see
> draft-bradner-restore-proposed (expired today but can be
> reposted if that would be useful)).
>
> So this draft now ignores the threshold problem for Proposed
> Standard and argues that "Changing the Internet Standards
> Process from three maturity levels to two is intended to create
> an environment where lessons from implementation and deployment
> experience are used to improve specifications".  But there is no
> logic to support the belief that intention will be satisfied.
> The purpose of the three-level system is to create an
> environment where the same lessons can be learned, applied, and
> documented.  This document does nothing to change or improve
> that environment unless one believes that, magically, removing
> the third level will increase motivations for getting things to
> draft.  The problem, which the document still describes, is that
> documents don't move off the first level, not that somehow the
> existence of the third one is pressing things down.
>
> In progressing by baby steps, actual babies fall down a lot.
> Usually they learn while falling and keep trying until they get
> it right.  Equally important, for their purposes, the important
> target is learning how to walk more reliably, not reaching a
> specific goal or solving a particular problem.  The IETF lacks
> those advantages: First, we are focused on the goal of making a
> change and have posited that change in terms of a choice between
> this particular document and doing nothing --all other
> possibilities have been dismissed without significant
> consideration.  To quote one of the co-authors of this draft
> from a different thread:
>
>> Herein lies the real problem:  As with many process and
>> structure discussions in the IETF, folk often see only a
>> simplistic, binary choice between whatever they prefer, versus
>> something akin to chaos.
>>
>> The world is more nuanced than that, and the choices more rich.
>
> I agree, but this document is being treated as that sort of
> binary choice.
>
> Second, the community is easily exhausted by process debates and
> the debate about this particular document has now been going on
> for 13+ months.  Even if the suggestion made at the plenary
> about opening 2026 were serious, does anyone actually believe
> the community will be able to take up and evaluate more
> significant process changes in the near future?  Or that this
> document, if approved, will not be used to justify deferring
> discussion of other changes because we have to see how it works
> out?
>
> So this is not "baby steps".  It is one step, a step that makes
> a change that isn't demonstrably connected to the problem it
> purports to solve and that leads into a dead end.
>
>     john
>
>
>>
>
>
>
>
>
>