Re: [ippm] Proposed charter updates for IPPM

["Carlos Pignataro (cpignata)" <[e-mail]>]

Thank you Brian for proactively bringing this discussion on the list before Prague.

All the proposed changes below bring the charter (i.e., the contract between the WG and the IETF) closer to reality and remove potential ambiguities and sources of misinterpretation. I believe that all these edits serve the higher purpose of charter hygiene and forward-looking polishing, and are not narrowly written solely with iOAM in mind.

As such, I very much support this proposal.

Related, and likely a question for Benoit, but curious about the PM-Directorate and whether this is OBE (I've not seen traffic for a couple years):
https://www.ietf.org/iesg/directorate/performance-metrics.html

I also agree with Al and support the removal (instead of rewrite) of the extra sentence.

Please find a couple small follow-ups inline.

Sent from my iPad

On Jul 9, 2017, at 1:23 PM, MORTON, ALFRED C (AL) <[e-mail]<mailto:[e-mail]>> wrote:

Hi Brian and Bill,

Thanks for preparing a new charter proposal and organizing
this message to highlight the changes. I agree with your
proposals.

I have one further proposal for the last sentence of
paragraph 1.

OLD (and your NEW):
Metrics developed by the IPPM WG are intended to provide unbiased
quantitative performance measurements and not a value judgement.

I propose to delete/replace the last phrase, which I think has been present
since the very first charter. IIRC, the concern was that contributions
might attempt to establish service acceptance thresholds, or other
numerical performance requirements/objectives using the metrics.
The work of setting numerical objectives proceeded elsewhere
(ITU-T Rec Y.1541), and the Earth continued to rotate.

If we want to ensure the status quo, we could say:
s/and not a value judgement/but numerical thresholds for the metrics are out-of-scope/

I also suggest this change with MBM in mind, where we have
testing outcomes of Pass/Fail/Inconclusive (PFI). Of course, the user
supplies the target rate, RTT, and/or loss ratio numerical values,
not the working group.

So we could express an explicit restriction that does not hamper MBM PFI:

NEW^2:
Metrics developed by the IPPM WG are intended to provide unbiased
quantitative performance measurements, but numerical thresholds for
the metrics are out-of-scope.

Or, we could remove the phrase, and refer any contributions on
numerical thresholds to ITU-T SG 12, Question 17, to avoid
overlapping work with another SDO (coordination with this SDO
should also be mentioned somewhere).

NEW^3:
Metrics developed by the IPPM WG are intended to provide unbiased
quantitative performance measurements.


+1 to this one.

thanks for considering this proposal, and regards,
Al

-----Original Message-----
From: ippm [mailto:[e-mail]] On Behalf Of Brian Trammell
Sent: Sunday, July 09, 2017 11:24 AM
To: IETF IPPM WG
Subject: [ippm] Proposed charter updates for IPPM

Greetings, all,

Bill and I met briefly last week to discuss a proposed change to the
charter, as a way to resolve the rough consensus to adopt the IOAM work
within IPPM with concerns that have been raised that that work doesn't
really fit on the charter. We arrived at a minimal suggested set of
edits. Though inspired by IOAM, these aren't so much focused on bringing
IOAM on charter as they are making the charter reflect the direction
work in IPPM has been going over the five years since we last updated
the charter.

We propose only to touch three paragraphs, as below:

---

OLD Paragraph 1:

The IP Performance Metrics (IPPM) Working Group develops and maintains
standard metrics that can be applied to the quality, performance, and
reliability of Internet data delivery services and applications running
over transport layer protocols (e.g. TCP, UDP) over IP. Specifying
network or lower layer OAM mechanisms is out of scope of the IPPM
charter. It also develops and maintains protocols for the measurement of
these metrics. These metrics are designed such that they can be used by
network operators, end users, or independent testing groups. Metrics
developed by the IPPM WG are intended to provide unbiased quantitative
performance measurements and not a value judgement.

NEW Paragraph 1:

The IP Performance Measurement (IPPM) Working Group develops and
maintains standard metrics that can be applied to the quality,
performance, and reliability of Internet data delivery services and
applications running over transport layer protocols (e.g. TCP, UDP) over
IP. It also develops and maintains methodologies and protocols for the
measurement of these metrics. These metrics are designed such that they
can be used by network operators, end users, or independent testing
groups. Metrics developed by the IPPM WG are intended to provide
unbiased quantitative performance measurements and not a value
judgement.


Two basic edits here:

(1) Rename working group from IP Performance Metrics to IP Performance
Measurement,

This is a very important change. Thanks for raising it.

and add "methodologies" to the set of things we do. In our
opinion, this reflects work that has been happening on charter for a
while: MBM, PDM, and alt-mark are all methodology work as opposed to
metric work.


I agree.

Editorial question: where it says: "These metrics are designed such that..." should that be: "These metrics, protocols, and methodologies are designed such that..."?

(2) Remove the sentence "Specifying network or lower layer OAM
mechanisms is out of scope of the IPPM charter". In my personal opinion,
this isn't really necessary to bring the IOAM work on, since though IOAM
can be carried on lower-layer headers, it doesn't really specify an OAM
mechanism as I understand that term to be defined in the routing area.
However, this restriction on the IPPM doesn't appear to do anything
useful beyond reinforcing an artificial silo dividing data-plane and
control-plane measurement, make it difficult for us to work together
with other (e.g. INT or RTG area) working groups on performance
measurement, and cause confusion as to whether we can do a thing called
"IOAM" in IPPM.

---

OLD Paragraph 6:

The WG has produced protocols for communication among test equipment to
enable the measurement of the one- and two-way metrics (OWAMP and TWAMP
respectively). These protocols will be advanced along the standards
track. The work of the WG will take into account the suitability of
measurements for automation, in order to support large-scale measurement
efforts. This may result in further developments in protocols such as
OWAMP and TWAMP. Agreement about the definitions of metrics and methods
of measurement enables accurate, reproducible, and equivalent results
across different implementations. To this end, the WG will define and
maintain a registry of metric definitions. The WG encourages work which
assesses the comparability of measurements of IPPM metrics with metrics
developed elsewhere.

NEW Paragraph 6:

The WG has produced protocols for communication among test equipment to
enable the measurement of the one- and two-way metrics (OWAMP and TWAMP
respectively). These protocols will be advanced along the standards
track. The work of the WG will take into account the suitability of
measurements for automation, in order to support large-scale measurement
efforts. This may result in further developments in protocols such as
OWAMP and TWAMP. Agreement about the definitions of metrics and methods
of measurement enables accurate, reproducible, and equivalent results
across different implementations. To this end, the WG defines and
maintains a registry of metric definitions. The WG encourages work which
assesses the comparability of measurements of IPPM metrics with metrics
developed elsewhere.


This is a purely editorial change: move definition and maintenance of
registry to present tense, since we're doing it.

---

OLD Paragraph 7:

The WG also encourages work which improves the availability of
information about the context in which measurements were taken.

NEW Paragraph 7:

The WG also encourages work which improves the availability of
information about the context in which measurements were taken, for
example, measurement implementation information, conditions on the
networks on which measurements are taken, or information about the data-
plane topology of the measured network.


This is an arguably editorial change. It makes it clear that work
targeted at IPPM that focuses on measurement on the spatial
characteristics of paths as context for metrics (e.g. parts of IOAM, but
also work like draft-amf-ippm-route, which attempts to answer the
question "what about traceroute" applied to the registry).


Good clarification.

The "for example" speaks to these, but I would add that context is not limited to this enumeration.

---

We additionally propose to strike the (out of date) near term milestones
from the charter text; milestones are separately managed.


Beyond these updates, we propose to leave the charter unchanged. We've
got some time to have this discussion on the agenda in Prague, but in
the meantime please send your thoughts on this proposal to the mailing
list.


Thanks!!!

Carlos.


Many thanks, best regards,

Brian (for the chairs)

_______________________________________________
ippm mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/ippm