[ippm] New Liaison Statement, "LS on harmonization of IP Capacity and Latency Parameters: Consent of Draft Rec. Y.1540 on IP packet transfer performance parameters and New Annex A with Lab & Field Evaluation Plans"

[Liaison Statement Management Tool <[e-mail]>]

Title: LS on harmonization of IP Capacity and Latency Parameters: Consent of Draft Rec. Y.1540 on IP packet transfer performance parameters and New Annex A with Lab & Field Evaluation Plans
Submission Date: 2019-05-22
URL of the IETF Web page: https://datatracker.ietf.org/liaison/1645/
Please reply by 2019-09-01
From: Judit Kiss <[e-mail]>
To: Brian Trammell <[e-mail]>,Bill Cerveny <[e-mail]>,Tommy Pauly <[e-mail]>
Cc: Scott Mansfield <[e-mail]>,Bill Cerveny <[e-mail]>,IP Performance Measurement Discussion List <[e-mail]>,Mirja Kühlewind <[e-mail]>,Brian Trammell <[e-mail]>,Magnus Westerlund <[e-mail]>,[e-mail],Tommy Pauly <[e-mail]>
Response Contacts: A. C. Morton <[e-mail] >
Technical Contacts: 
Purpose: For action

Body: Further Progress on Evaluating IP Capacity Metrics and Methods for Revised Recommendation Y.1540, “Internet protocol data communication service – IP packet transfer and availability performance parameters”

As an update to our communications from the Q17/12 Interim meeting (Darmstadt, 16-17 October 2018), the Geneva, 27 November – 6 December 2018 meeting with the full SG12, and the results from the subsequent Q17/12 Interim meeting (Berlin, 5-7 March 2019), we now share further results from the Q17/12 meeting with the Full Study Group 12 (Geneva, 7-16 May 2019). 
We have completed all planned tests using the Phase 1 Laboratory test bed that supports development our revised Rec. Y.1540 and new Annex A/Y.1540 specifications of IP Capacity and Latency methods of measurement. We have compared several existing measurement methods based on TCP and UDP transport. The Phase 2 Network tests on live networks are also sufficiently complete for our work to proceed.

Our key results and cumulative findings from the May 2019 Geneva meeting are:

•	The Survey of prior testing and academic publications is now complete, finding that: 
o	The benchmark measurement of the maximum achievable access speed is only obtained by flooding the access link with UDP traffic. A large scale research survey and the test tool of a major content provider prefer UDP to accurately measure access speed.
o	Parallel TCP based measurements can only be used to estimate Internet access bandwidth, and several publications discuss specific factors reducing the accuracy of parallel TCP based measurements. TCP based measurements fail to detect the presence of background traffic, causing further issues and inaccuracies. 
o	In a recent study, UDP was found to be a viable Internet transport protocol, based on tests in North & South America, Europe, Asia and Oceania. UDP traffic was blocked on some ports or entirely in rare cases. 
o	Material on WiFi performance and Encrypted stream Network QoS parameters introduce a set of KPIs/metrics prioritized through machine learning, and models of QoE.  
•	The text of the IP Capacity definition and Methods of measurement, and the Phase 1 and Phase 2 sections of the test and evaluation plan were edited and agreed during the meeting. In the agreed text, the Phase 1 study was a Lab evaluation, and Phase 2 study was carried-out in production networks. This material constitutes revisions to Y.1540 sections and the new ANNEX A/Y.1540. The Q17 meeting agreed to seek Consent on this material.
•	The meeting agreed on an expanded Summary of the Phase 1 and Phase 2 Testing as a new informative APPENDIX to Y.1540, to provide the supporting test data for the conclusions represented as requirements in the body and Annex A. Some conclusions are:
o	Phase 1/Lab: Even without correction factors, the UDP measurements are considerably closer to the calibrated shaper rate than TCP, despite using the most favourable circumstances for TCP (no added delay or background traffic). With correction for headers that are viewed by the traffic shaper, UDP measurements using iPerf 2 are within 200ppm of the configured shaper rate. UDP-based measurements are the Benchmark for capacity, accurately assessing the “ground truth” of the traffic shaper rate under all tested conditions.
o	Phase 1/Lab: TCP measurements using iPerf2 underestimate the shaper rate with or without correction factors. Typical round-trip delay and the presence of competing/background traffic tend to make TCP-based estimates of available capacity appreciably worse, and introduce considerable results variability that reduces confidence in any measurement.
o	Phase 2/Field: The Phase 1/Lab conclusions on UDP-based testing as the Benchmark, and TCP as underestimating capacity were supported by the Phase 2/Field measurements. TCP measurements on 1 Gbps PON exhibit significant underestimation of capacity. UDP-based methods are clearly the preferred transport protocol for Capacity assessment.
•	It is clear from on-going measurements that Internet subscriber use of TCP protocol is declining. Video and browser traffic has been shifting to UDP and higher-layer reliability mechanisms for years, with the most significant growth in 2018 due to CDN adoption. 

SG12 continues to invite interested parties to validate testing results, and to coordinate on the development of a new generation of harmonized specifications of IP Capacity performance metrics and methods of measurement, and other key performance parameters.

In one year’s time, Study Group 12 has completed many new areas of investigation, Consented its key IP performance Recommendation (Y.1540) with the newest Capacity metrics, and reached the point where development of coordinated specifications can proceed among collaborating SDOs.  

All Q17/12 participants continue to seek additional SDO and individual support, participation and/or constructive review, so that together, we can proceed forward.
Attachments:

    oLS-92
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-05-22-itu-t-sg-12-ippm-ls-on-harmonization-of-ip-capacity-and-latency-parameters-consent-of-draft-rec-y1540-on-ip-packet-transfer-perfo-attachment-1.docx

    oLS-93
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-05-22-itu-t-sg-12-ippm-ls-on-harmonization-of-ip-capacity-and-latency-parameters-consent-of-draft-rec-y1540-on-ip-packet-transfer-perfo-attachment-2.docx

    oLS-94
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-05-22-itu-t-sg-12-ippm-ls-on-harmonization-of-ip-capacity-and-latency-parameters-consent-of-draft-rec-y1540-on-ip-packet-transfer-perfo-attachment-3.docx