Re: [arch-d] draft-iab-rfc7500-bis-00 - new principle of measureability

[Jay Daley <[e-mail]>]

Hi John

Thank you for such a detailed and thoughtful reply.

Rather than address each point, let me just note one thing, which is that the absence of this principle doesn’t stop the problems you outline below. Metrics are still set, expectations still formed and measurements still made.  Without a principle that is generally very narrowly defined as a set of contractual requirements and so becomes a pull of data for contractual purposes only rather than a principled push of data for use as the community wishes.

However, you’re quite right that this proposed principle doesn’t have anything close to consensus and partly I think that’s because the language I used fell into the same trap of constraining the measurement for the primarily 'contractual' purpose of assessing performance, ("performance" being in itself a contentious word).

I still think there's a need for a principle that requires data collection and open publication, and if we could leave it at "in an open, transparent, accountable and *measureable* way" then that would do it for me, but it’s hard to see how that can be detailed further while avoiding the issues.

kind regards
Jay

> On 28/06/2019, at 11:35 AM, John C Klensin <[e-mail]> wrote:
> 
> 
> 
> --On Thursday, June 27, 2019 12:08 +1200 Jay Daley
> <[e-mail]> wrote:
> 
>> Thanks Brian
>> 
>> I was trying to keep this strictly at the principle level and
>> a gap I see in the proposed principles, rather than get into
>> any details of how performance expectations are set or
>> measured.
> 
> Jay,
> 
> Please take this as a comment about the principle and not about
> this particular I-D -- I'd be very opposed to altering the
> document beyond the current scope of changes even if there were
> agreement on your proposed principle, which, at least IMO, there
> isn't.
> 
> There is a long and unfortunate history about how efforts to
> quantify and measure complex administrative procedures (or
> complex procedures with significant administrate components)
> tend to work out.   There are inevitably some things that are
> easy to quantify and measure.  Using IANA as an example,
> consider registration lag (time between the arrival of a
> registration request and its appearance in the relevant
> registry), registrations completed per month, and so on.  Other
> things are much harder to measure.  For example, requests
> sometime come along that ask IANA to reorganize a registry.
> While one could count the number or frequency of such requests,
> that would ultimately measure the behavior of the body making
> the requests -- the only thing it measures about IANA is their
> ability to read minds and forecast the future, neither of which
> should be anything we try to write into a contract (I hope for
> obvious reasons).  But what they happens in practice is that the
> things that are easy to quantify and measure get redefined as
> being _the_ performance criteria and the other things tend to
> get submerged and lost.  
> 
> That can be pathological because it may reward bad behavior and
> discourage good behavior.   For example, we have "IANA
> Considerations" sections in documents so IANA can review what
> they are being asked to do before the IETF finalizes the
> request.   Obviously, we don't want them to take any longer to
> make those evaluations than necessary.  But it is far more
> important that the make them with great care and as much thought
> and dialogue as needed.  If they are encouraged (or forced) to
> race through things by being evaluated based on throughput and
> don't spot important issues as a result, they lose, the IETF
> loses, and the users of the resulting registries and database
> lose most of all.
> 
> So I suggest that trying to enshrine "let's find things to
> measure and take those measurements" as a principle -- and it is
> where "include the collection and publication of sufficient data
> ...to allow their performance ...to be measured and to assess
> whether they are delivering their services" has a long history
> of taking us and others -- is, by itself, a bad idea.
> 
> Now those concerns about misuse of quantitative data and
> effective discarding of qualitative information can be mitigated
> by deep understanding of issues, careful design, implementation,
> and oversight so I'm not arguing against data collection under
> the right set of circumstances.  But, unless whatever data
> collection and measurement plans are developed considers the
> issues, risks, and controls --things that generally goes far
> beyond (and are orthogonal to) openness and transparency -- they
> are often more of a problem than an advantage.
> 
> If you want to write an Internet-Draft that proposes what should
> be measured in the IANA case _and_ that provides a context for
> proper consideration of the qualitative factors and protection
> against rewarding them for rushing through things rather than
> doing them carefully and with due consideration, I, for one,
> would be very interested in seeing it.  Probably so would the
> IANA staff as well as the IAB.  But the "principle of
> measurability" as proposed above is, independent of anything
> else that has been said in this thread, just not a good idea.
> 
> best,
>    john
> 
> 
> 
> 

-- 
Jay Daley
[e-mail]
[phone]
skype: jaydaley
twitter: @jay_daley