Re: [ippm] New Liaison Statement, "LS - Harmonization of IP Capacity and Latency Parameters: Revision of Draft Rec. Y.1540 on IP packet transfer performance parameters and New Annex A with Lab Evaluation Plan"

["MORTON, ALFRED C (AL)" <[e-mail]>]

IPPM,

Following a side-meeting discussion of the Subject Liaison,
and several individuals indicating continuing interest
in this work,  I would like to propose the brief text of
a Liaison reply. 

The WG will need to reach consensus on the text of
the reply as a next step.

Thanks to all who have expressed interest in this work!
regards,
Al

Proposed Body of Liaison Reply (to all recipients) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

To: ITU-T SG12, ITU-T SG11, BBF ETSI TC STQ, ETSI TC INT

IETF IPPM working group thanks you for your liaison describing results from
your evaluation of IP Capacity and Latency Metrics and  methods of measurement. 
The harmonization of these measurements is welcome in our industry.

We have the following comments on the preliminary results, and
the evaluation plan:

After noting the efficacy of UDP measurements in your results,
we can state that UDP transport is the basis for most IPPM measurement
implementations, and IPPM's measurement protocols.

When assessing IP Capacity and Latency, there should be 
no attempt to determine the technology involved in the path, 
such as AQM. The method should measure the path as a black box. 

The presence of packet-marking-sensitive technologies, such as
Diff-serv queues add complexity to IP Capacity and Latency.

There is not yet a metric that characterizes the BW-Delay product
in the IPPM-literature. 

Live network testing may be affected by Network operator policy,
it could change from the lab measurements - the test traffic is
part of the background traffic. OTOH, the operator could be 
prioritizing test traffic, especially their own authorized testing.

We also suggest to include QUIC-based measurements in your Lab 
evaluation, if possible.

Where a model or post-processing needs to be applied to the measurements,
this aspect must also be specified, especially for transport with multiple 
connections (averaging the connection performance).

Some of the IPPM WG participants who indicated their interest in
November 2019 have begun to share their comments informally, 
as planned.

Please keep the IPPM WG informed of your progress, especially if there are
ways in which IPPM WG participants assist further.

<signed by chairs of IPPM WG>


> -----Original Message-----
> From: Liaison Statement Management Tool [mailto:[e-mail]]
> Sent: Wednesday, March 13, 2019 1:23 PM
> To: Bill Cerveny <[e-mail]>; Brian Trammell <[e-mail]>;
> Tommy Pauly <[e-mail]>
> Cc: Mirja Kühlewind <[e-mail]>; Scott Mansfield
> <[e-mail]>; Bill Cerveny <[e-mail]>; MORTON,
> ALFRED C (AL) <[e-mail]>; Spencer Dawkins
> <[e-mail]>; IP Performance Measurement Discussion
> List <[e-mail]>; Brian Trammell <[e-mail]>; itu-t-
> [e-mail]; Tommy Pauly <[e-mail]>
> Subject: New Liaison Statement, "LS - Harmonization of IP Capacity and
> Latency Parameters: Revision of Draft Rec. Y.1540 on IP packet transfer
> performance parameters and New Annex A with Lab Evaluation Plan"
> 
> Title: LS - Harmonization of IP Capacity and Latency Parameters: Revision
> of Draft Rec. Y.1540 on IP packet transfer performance parameters and New
> Annex A with Lab Evaluation Plan
> Submission Date: 2019-03-13
> URL of the IETF Web page:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_liaison_1632_&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=uLz_oPVMGlxwhg8pC1vN9Gw4ypYFZ
> k506Axq2RLOyo0&s=ynxvwzaXF2tlJ0iSuBqmqYiJfPueJhlEBFtiyfxfNcg&e=
> Please reply by 2019-05-01
> From: Judit Kiss <[e-mail]>
> To: Brian Trammell <[e-mail]>,Bill Cerveny
> <[e-mail]>,Tommy Pauly <[e-mail]>
> Cc: Scott Mansfield <[e-mail]>,Bill Cerveny
> <[e-mail]>,Brian Trammell <[e-mail]>,IP Performance
> Measurement Discussion List <[e-mail]>,Mirja Kühlewind
> <[e-mail]>,Spencer Dawkins <[e-mail]>,itu-
> [e-mail],Tommy Pauly <[e-mail]>
> Response Contacts: A. C. Morton <[e-mail]>
> Technical Contacts:
> Purpose: For action
> 
> Body: As an update to our communications from the Q17/12 Interim meeting
> (Darmstadt, 16-17 October 2018) and the Geneva, 27 November - 6 December
> 2018 meeting with the full SG12, we now share further results from the
> Q17/12 Interim meeting (Berlin, 5-7 March 2019).
> 
> We have completed nearly all planned tests using the Phase 1 Laboratory
> test bed that supports development our revised Rec. Y.1540 and new Annex
> A/Y.1540 specifications of IP Capacity and Latency methods of measurement.
> We have compared several existing measurement methods based on TCP and UDP
> transport. The Phase 2 Network tests are just starting now.
> 
> We continue to invite interested parties to join the testing efforts, and
> to coordinate on the development of a new generation of harmonized
> specifications of IP Capacity performance metrics and methods of
> measurement, and other key performance parameters.
> 
> Our key results and current findings from the Berlin meeting are:
> 
> •	The text of the IP Capacity definition and Methods of measurement,
> and the Phase 1 and Phase 2 sections of the test and evaluation plan were
> edited and agreed during the meeting. In the agreed text, the Phase 2
> study will be carried-out in production networks, and lab networks if they
> are made available in a timely fashion. This material constitutes
> revisions to Y.1540 sections and the new ANNEX A/Y.1540. The meeting
> agreed to seek Consent on this material in May, 2018.
> •	The meeting agreed to prepare the Preliminary Test Summary
> (presented for the first time at this meeting) as a new informative
> APPENDIX to Y.1540, to provide the supporting test data for the
> conclusions represented as requirements in the body and Annex A.
> •	Even without correction factors, the UDP measurements are
> considerably closer to the calibrated shaper rate than TCP, despite using
> the most favourable circumstances for TCP (no added delay or background
> traffic). With correction for headers that are viewed by the traffic
> shaper, UDP measurements using iPerf 2 are within 200ppm of the configured
> shaper rate. UDP-based measurements are the benchmark for capacity,
> accurately assessing the “ground truth” of the traffic shaper rate under
> all tested conditions.
> •	TCP measurements using iPerf2 underestimate the shaper rate with or
> without correction factors. Typical round-trip delay and the presence of
> competing/background traffic tend to make TCP-based estimates of available
> capacity appreciably worse.
> •	Another contribution updated the overall academic Survey, adding
> material to one section on WiFi performance and one new section Encrypted
> stream Network QoS parameters and their relationship to QoE. Both topics
> introduce a set of KPIs/metrics prioritized through machine learning, and
> models of QoE.  The relevance to active testing includes the possible
> additional metrics collected, or input for stream design.
> •	It is clear from the surveys presented that Internet subscriber use
> of TCP protocol is declining. Video and browser traffic has been shifting
> to UDP and higher-layer reliability mechanisms for years, with the most
> significant growth in 2018 due to CDN adoption.
> •	There was a demonstration of a new prototype measurement tool that
> makes UDP-based Capacity, Latency, and Loss measurements while searching
> for the peak Capacity that can be supported on the tested path. The
> meeting provided useful feedback for the developer.
> 
> Q17/12 and collaborating SDOs have covered many new areas of
> investigation, and are approaching the point where development of
> coordinated specifications can proceed.
> All group members continue to seek additional SDO and individual support,
> participation and/or constructive review, so that together, we can proceed
> toward the next generation of IP performance metrics.
> 
> Attachments:
> –	SG12-TD775 with revised Y.1540 clauses and new Y.1540 Annex A;
> –	SG12-TD776 with new Y.1540 Appendix, Phase 1 Test Summary.
> Attachments:
> 
>     TD775
>     https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2019-2D03-2D13-2Ditu-
> 2Dt-2Dsg-2D12-2Dippm-2Dls-2Dharmonization-2Dof-2Dip-2Dcapacity-2Dand-
> 2Dlatency-2Dparameters-2Drevision-2Dof-2Ddraft-2Drec-2Dy1540-2Don-2Dip-
> 2Dpacket-2Dtransfer-2Dperfo-2Dattachment-2D1.docx&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=uLz_oPVMGlxwhg8pC1vN9Gw4ypYFZ
> k506Axq2RLOyo0&s=dAR1A1k2C0NFX677nghi88SGItQwZxXCUB3dnMtgshw&e=
> 
>     TD776
>     https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2019-2D03-2D13-2Ditu-
> 2Dt-2Dsg-2D12-2Dippm-2Dls-2Dharmonization-2Dof-2Dip-2Dcapacity-2Dand-
> 2Dlatency-2Dparameters-2Drevision-2Dof-2Ddraft-2Drec-2Dy1540-2Don-2Dip-
> 2Dpacket-2Dtransfer-2Dperfo-2Dattachment-2D2.docx&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=uLz_oPVMGlxwhg8pC1vN9Gw4ypYFZ
> k506Axq2RLOyo0&s=S0LEXw6CpSLFV3ME16BU0wA4iUoUEjs5gLFLs6ElumA&e=