Re: [ietf-outcomes] "Instances" column

["David Harrington" <[e-mail]>]

Hi, 

> -----Original Message-----
> From: Dave CROCKER [mailto:[e-mail]] 
> Sent: Sunday, February 14, 2010 1:55 PM
> To: David Harrington
> Cc: [e-mail]
> Subject: Re: [ietf-outcomes] "Instances" column
> 
> David,
> 
> 
> On 2/14/2010 10:21 AM, David Harrington wrote:
> > Hi Dave,
> >
> > I would like to see more consistency with existing IETF
terminology.
> >
> > RFC2026 defines three levels of standard.
> > Proposed requires an agreed upon document.
> > Draft requires implementations.
> > Standard requires operational experience.
> 
> Well, that's certainly a worthy discussion and it would be 
> great to get the list 
> engaged on it.
> 
> And indeed, I'd guess that ++ and Full Standard ought to 
> correlate highly.

Maybe ought to, but often do not.
SNMPv3 is a full standard, but doesn't achieve ++ usage.
The IETF decided years ago not to bother taking MIB modules to draft
and full standards because most vendors implement the Proposed
Standard, if it is applicable to their products. A number of MIB
modules might achieve ++
usage but remain at PS. Other MIB modules also are at PS, but achieve
no usage.

So I recognize that the formal and informal adoption levels can be
very different.
I am not suggesting that we want the wiki to reflect the formal
status.
I think it can be important to understand that formal status and
informal status can differ markedly.
Maybe we should understand where and why, so we can change the formal
process to better match reality.
(That work, of course, has been underway I think since 2026 was
published ;-)

> 
> There is an underlying question that I'd suggest you post an 
> answer to:
> 
>        Why should the two be linked?

Most comments to this list have been about the confusing terminology
related to adoption.
If we were only informally measuring adoption, then I would not argue
that we should use existing IETF terminology.
But when you add instances, you are now talking informally about both
implementation and operational experience.
If we are going to have both of these metrics in the wiki, which
happen to be roughly the same metrics we use for standards, then I
think we should use consistent terminology. I think the adoption
levels are confusing enough, but if you are now going to discuss
instances as well as adoption levels, you seem to be adding
implementation information. And I think that confuses what adoption
means - SNMPv3 was (initially) widely implemented in devices, but not
in managers, so it didn't get used. So how do you measure adoption? Do
many instances imply ++ adoption?

That's the big issue I have. I think the terminology being used is
confusing to interpret because there are mixed metrics, and adding
'instances' seems to confuse it even more.

If you think of the formal process as "two interoperable
implementations" and "scalability" rather than just "implementations"
and "usage", then I agree the two are not the same and should not be
linked. 

> 
>        What benefit derives?

The main benefit I see is that engineers understand the distinction
between implementation and usage, and the overwhelming majority of the
readers of this wiki will have an engineering bent. 
I think instances and adoption lacks clarity.


> 
>        On what basis would it make sense for them /not/ to be
linked?
> 
> For example, the standards label is a formal, political 
> process.  It's rare and 
> difficult to get to Full.  (The YAM wg has embarked on a test 
> of this, and the 
> experience so far is not pretty.)
> 
> The wiki is intended as an extremely informal and flexible process.
> 
> The difference between these is fundamental.
> 
> 
> > RFC2026 doesn't talk about instances and usage, it talks about
> > implementations and operational experience.
> 
> For Draft it is only required to talk about /two/ implementations.
> 
> Full Standard pretty much /is/ in terms of usage.
> 
> 
> > Your "instances" column is about implementations, but uses
different
> > terminology than is already in use in the IETF.
> 
> If your concern is 'instance' vs. 'implementation', that's to 
> keep the column 
> narrower.  And no, I'm not kidding.  If there is a real 
> indication of community 
> misunderstanding, then the disparity is important.  However 
> the idea that the 
> community cannot tolerate synonyms -- especially given how 
> loose people tend to 
> be with their vocabulary -- should be documented.
> 
> If your concern is something else, please elucidate.
> 
> 
> > Your "adoption" column is about operational experience, but uses
> > different terminology than IETF process.
> 
> What do you believe is established terminology that achieves 
> the same purpose as 
> the Adoption column is trying to serve?

operational experience.
I'm fine with usage.

> 
> 
> > I would like to see the results wiki use terminology consistent
with
> > our standards process.
> 
> That's an inclination that has intuitive appeal.
> 
> However we need to be careful that it does not produce 
> slavish adherence that 
> undermines actual utility.  Since the standards process and 
> the wiki are not 
> identical instruments, it's entirely possible they need 
> disparate terminology.
> 
> I'm not saying I'm sure they do, but merely that the decision 
> needs to be made 
> carefully?
> 
> 
> > I think using terminology consistent with our process would 
> make these
> > adoption categories much more intuitive to IETF participants, AND
to
> > those who follow the IETF, such as journalists.
> 
> Again, that has intuitive appeal.  Note, however, that 
> essentially everyone but 
> serious, longer-term IETF geeks have no feel for the 
> standards levels at all. 
> So I am not sure I see how the link makes anything intuitive 
> for the larger 
> community.  (FWIW, I'm not happy about that observation, but 
> I think it's 
> factually correct.)

I did not say we should use terms like Proposed, Draft and Full
Standard which most people who haven't been in the IETF for a long
time would not necessarily understand. But I think engineers would
understand the distinction between implementation and operational
experience (or usage) better then between instances and adoption.

> 
> 
> > I think even to an
> > IETF newbie, the engineering distinction between implementation
and
> > operational experience would be clear; I do not think the 
> distinction
> > between instances and adoption is very clear.
> 
> You still have not answered the underlying question I've 
> asked a couple of times:
> 
>     Implementation is interesting only for standards that 
> have not been very 
> successfull.  As such it is a secondary measure.

But isn't instances a pointer to implementations?
If not, then I guess I really don't know what 'instances' means or
what having it here gives me.
I think instances and implementations are synonymous, although
implementations is more specific (and thus narrower).
Please explain how instances is narrower than implementations.

As you observed, a listing of 'instances' tends to promote the
relevant standard, presumably to improve usage.
Does that mean that listing instances is interesting only for
standards that have not been very successful?
Or are we listing instances to help market products?
I'm not sure of the purpose of 'instances'.

> 
>     Why elevate it to primary?

I am suggesting elevating it to an even level with usage to help
better understand the curve of adoption. Implementation necessarily
precedes usage.

There might well be interesting correlation of timelines of
implementation versus usage.
BEEP doesn't get used, largely because there are very few
implementations. It is reputedly expensive to implement.
It has implementation issues. It's not used because it's not
available.
SNMPv3 hasn't gotten ++ usage, even though there are ++
implementations (on the agent side).
As somebody mentioned in this wiki discussion, SNMPv3 didn't get used
because configuring it is too hard. It's not that the implementations
aren't there; it has usage issues.
There are multiple successful standards that have both properties -
available implementations and easy to use.

I think it important to understand both implementation adoption (i.e
availability from vendors, which might include open source developers)
and usage adoption (by end-users) of a protocol. If the purpose of the
wiki is to allow us to mine the data for what makes a protocol a
notable success or failure, I think the availability and usage metrics
are far more relevant to success/failure than the number of years it
took to produce, or (in most cases) the year in which it was first
published.

> 
> 
> > It would also be nice if somewhere in the
"implementation/instances"
> > page hierarchy, we had pointers to BCP 9 (RFC 5657) "Guidance on
> > Interoperation and Implementation Reports for Advancement to Draft
> > Standard", and to specific implementation reports found at
> > http://www.ietf.org/iesg/implementation-report.html. This might
help
> > encourage implementers to file actual implementation reports.
> 
> David, you keep wanting to turn the wiki into an extension of 
> formal processes, 
> when it is carefully designed to be quite different from it.
> 
> We have extensive community assessment that the formal 
> process is problematic in 
> effort and vagaries.  The wiki is designed to be informal, 
> simple and direct. 
> It asks only one question:  Did an expensive community effort
succeed?
> 
> For each change you suggest, please think hard about the 
> value proposition of 
> the change and look for ways that an otherwise-reasonable 
> value proposition 
> might actually /not/ be true.

Well, part of the problem is that I do not really understand the
purpose of the wiki if there is an instances column.
Is it still about general rough perceptions of adoption, and instances
imply some level of adoption?
or is it about marketing, which the IETF typically stays away from?
I know Ray asked for instances. Why does he want that? Is he going to
buy an instance of the standard? 
Is it to prove that adoption is somehow reflected in the number of
products available? 
Is it to reflect the informal "maturity" of the standard in the
industry?

I think the otherwise-reasonable value proposition of the instances
column might actually not be very clear.
I think 'instances' can add to the confusion of the metrics used to
informally measure success and failure.
I think columns to represent implementations (vendor adoption) and
usage (user adoption) would be more useful than instances and
adoption.

> 
> d/
> -- 
> 
>    Dave Crocker
>    Brandenburg InternetWorking
>    bbiw.net
>