Re: [arch-d] draft-iab-rfc7500-bis-00 - new principle of measureability

[John C Klensin <[e-mail]>]

--On Thursday, June 27, 2019 12:08 +1200 Jay Daley
<[e-mail]> wrote:

> Thanks Brian
> 
> I was trying to keep this strictly at the principle level and
> a gap I see in the proposed principles, rather than get into
> any details of how performance expectations are set or
> measured.

Jay,

Please take this as a comment about the principle and not about
this particular I-D -- I'd be very opposed to altering the
document beyond the current scope of changes even if there were
agreement on your proposed principle, which, at least IMO, there
isn't.

There is a long and unfortunate history about how efforts to
quantify and measure complex administrative procedures (or
complex procedures with significant administrate components)
tend to work out.   There are inevitably some things that are
easy to quantify and measure.  Using IANA as an example,
consider registration lag (time between the arrival of a
registration request and its appearance in the relevant
registry), registrations completed per month, and so on.  Other
things are much harder to measure.  For example, requests
sometime come along that ask IANA to reorganize a registry.
While one could count the number or frequency of such requests,
that would ultimately measure the behavior of the body making
the requests -- the only thing it measures about IANA is their
ability to read minds and forecast the future, neither of which
should be anything we try to write into a contract (I hope for
obvious reasons).  But what they happens in practice is that the
things that are easy to quantify and measure get redefined as
being _the_ performance criteria and the other things tend to
get submerged and lost.  

That can be pathological because it may reward bad behavior and
discourage good behavior.   For example, we have "IANA
Considerations" sections in documents so IANA can review what
they are being asked to do before the IETF finalizes the
request.   Obviously, we don't want them to take any longer to
make those evaluations than necessary.  But it is far more
important that the make them with great care and as much thought
and dialogue as needed.  If they are encouraged (or forced) to
race through things by being evaluated based on throughput and
don't spot important issues as a result, they lose, the IETF
loses, and the users of the resulting registries and database
lose most of all.

So I suggest that trying to enshrine "let's find things to
measure and take those measurements" as a principle -- and it is
where "include the collection and publication of sufficient data
...to allow their performance ...to be measured and to assess
whether they are delivering their services" has a long history
of taking us and others -- is, by itself, a bad idea.

Now those concerns about misuse of quantitative data and
effective discarding of qualitative information can be mitigated
by deep understanding of issues, careful design, implementation,
and oversight so I'm not arguing against data collection under
the right set of circumstances.  But, unless whatever data
collection and measurement plans are developed considers the
issues, risks, and controls --things that generally goes far
beyond (and are orthogonal to) openness and transparency -- they
are often more of a problem than an advantage.

If you want to write an Internet-Draft that proposes what should
be measured in the IANA case _and_ that provides a context for
proper consideration of the qualitative factors and protection
against rewarding them for rushing through things rather than
doing them carefully and with due consideration, I, for one,
would be very interested in seeing it.  Probably so would the
IANA staff as well as the IAB.  But the "principle of
measurability" as proposed above is, independent of anything
else that has been said in this thread, just not a good idea.

best,
    john