[bmwg] FW: New Liaison Statement, "LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG11-LS52)"

["MORTON, ALFRED C (AL)" <[e-mail]>]

BMWG, 

This Liaison is of interest to the WG.
https://datatracker.ietf.org/liaison/1602/

There is some cross-over between measurement methods
we are using now in lab benchmarking, and possible 
production network measurements of the future.

We will spend a small amount of agenda time on this
topic during our session at IETF-103.

regards,
Al
bmwg co-chair

-----Original Message-----
From: Liaison Statement Management Tool [mailto:[e-mail]] 
Sent: Friday, October 26, 2018 2:54 PM
To: The IETF Chair <[e-mail]>
Cc: The IESG <[e-mail]>; The IETF Chair <[e-mail]>; [e-mail]; MORTON, ALFRED C (AL) <[e-mail]>; Scott Mansfield <[e-mail]>
Subject: New Liaison Statement, "LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG11-LS52)"

Title: LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG11-LS52)
Submission Date: 2018-10-26
URL of the IETF Web page: https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_liaison_1602_&d=DwIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=VX3GNCspVy25DZ0Zvm0i0HhrSCNzJBOHkhMrvjqcqlw&s=HCwQVX1oNj6zFdiNQ3fj1Onc-HW1gC7ZIOOdxqOrBYg&e=

From: Al Morton <[e-mail]>
To: The IETF Chair <[e-mail]>
Cc: Scott Mansfield <[e-mail]>,The IETF Chair <[e-mail]>,[e-mail],The IESG <[e-mail]>
Response Contacts: A. C. Morton <[e-mail]>
Technical Contacts: 
Purpose: For information

Body: ITU-T Study Group 12 thanks you for your liaison, sharing your plan to discuss Draft Q.3961, “Testing methodologies of Internet related performance measurements including e2e bit rate within the fixed and mobile operator’s networks”, during future workshops and Joint meetings. We are aware that text similar to draft requirements in Q.3961 was completely deleted from the corresponding ETSI INT specification as a result of extensive comments during the approval process, and there are additional comments post-approval which remain un-addressed and unresolved after several months. We have been notified of new and urgent work in ETSI STQ to revise clause 6 of TS 103 222-1 (the reference replacing almost all requirements in the ETSI INT specification). The current text of TS 103 222-1 (3-2018) specification retains a completely non-compatible scope to draft Q.3961, and suffers from the same shortcomings raised in the ETSI INT review process. As part of the Harmonization efforts
  for IP-network performance specifications between ETSI STQ and Q17/12, we have reviewed the revised text of clause 6/103 222-1, and find it satisfactory. Our conclusion on this topic is that we agree and support the request of ETSI STQ, to avoid referencing the (current) text of TS 103 222-1 (3-2018) while this work is in-progress. The work is proceeding toward a prompt resolution.

Study Group 12’s plan is to prepare relevant normative specifications for IP-layer Capacity and Flow-related Parameters (throughput), and that SG 11 make use of the new specifications when available, as communicated in May 2018.  We are pleased SG11’s recent liaison did not object to this plan, and we hope SG11 will participate in Harmonization efforts, as ETSI STQ has agreed.

As stated in your liaison, ITU-T SG11 envisages that benchmarking testing is one of the components of draft Recommendation ITU-T Q.3961. Since the term “benchmarking” is not defined in the liaison, and has many uses throughout the industry, we would like to understand what this form of benchmarking is intended compare. 

We are also aware of a list of methods used by Regulators to *survey* Internet performance, published by the OECD https://urldefense.proofpoint.com/v2/url?u=http-3A__www.oecd.org_internet_speed-2Dtests.htm&d=DwIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=VX3GNCspVy25DZ0Zvm0i0HhrSCNzJBOHkhMrvjqcqlw&s=-0O-ljmiu87BD6Xe7tWtTFujYxTklUIXzfmAw8v0ze8&e= . Some Regulators have made disclaimers about the accuracy of their survey results. For example:

        The locations and technical features of the broadband connections have been provided by the service users themselves, who are entirely and exclusively liable for their 
        accuracy. The speed resulting from the measurement as the final speed of the connection is also affected by a series of factors, such as the quality of cables, connections 
        and equipment or electromagnetic interference.

This well-expresses our concerns with the methods described in current SG 11 draft of Q.3961.

At this time, we would like to inform you of our progress to develop the aforementioned normative specifications in Study Group 12, especially our progress at the October 2018 Q17 Interim meeting in Darmstadt.

•	AT&T’s contribution describing how network access, usage patterns, user application requirements, and future measurement paths have changed in the past five years set the tone for the meeting. There are likely to be “Gigabit islands” at the edge of future networks. 
•	ETSI TR103501 has been presented and discussed. Essential points were the requirement for full transparency of the measurement and data processing methodology. The TR attempts to provide a full view of today’s methods for application level measurement including multi socket and crowd sourcing approaches.
•	Y.1540 IP capacity and flow related parameter clauses were edited by the group.
•	Y.1540 annex A was expanded to include two phases of testing. The first phase is a test using shaper/policer capabilities of a general purpose computing system, based on AT&T’s contribution and preliminary test results. This phase is parallel to the BEREC evaluation of Internet access measurements, and adding critical test path parameters.
•	Y.1540 annex A was expanded to include information on evaluation of different access speed measurement methods, conducted in 2012 by Goga and Teixeira. This work parallels our plan for phase 2.
•	There is a clear need to enlist active participants in the testing program, who are willing to contribute both resources and personnel consistent with the plans under development. Those who choose not to participate will have no influence on the outcome, or the schedule for completion, so they are encouraged to participate or wait patiently.  
•	A list of regulator Internet access survey methods was compiled by AT&T and discussed during the meeting. This list is more up to date than the 2015 OECD list, and concludes that there are many different methods currently in use, and no International consensus can be derived to support standard development from reading such lists.
•	AT&T contributed a comparison of two methods of LTE Internet access speed assessment. The results show wide variation and strong dependence on mobile device type. Larger sample sizes are needed to achieve the comparison goals.
•	Q17 reviewed the ETSI-STQ’s revised clause 6 of TS-103 222-1, from the rapporteur of this new work item. The text was found to be agreeable, with the suggestion to update the introduction to indicate these changes.

Study Group 12 has provided a detailed explanation of the clause 6.12 requirements in Appendix IX of Rec. Y.1540, titled “Explanation of TCP-based measurement inadequacy to meet normative requirements”.  However, we have concluded that Capacity measurements described in RFC 8337 meet the requirements in clause 6.12 of ITU-T Recommendation Y.1540. 

We invite all experts to participate in the evaluation plan’s development and to subsequently participate in the evaluations. SG 11’s plans to develop the text of the draft Q.3961 should now move forward after the evaluation and approval of Study Group 12’s normative specifications are complete, according to the contribution-driven schedule.  Certainly, SG 11 experts will be willing to contribute their expertise and resources to this plan, consistent with their desire to complete work on this topic as quickly as possible.

The current draft of revisions to Y.1540, and a new Annex A for the new parameter definitions now contains the current text of the evaluation plan for candidate definitions for the new normative parameters for IP-layer Capacity and Flow-related Parameters (throughput). Annex A contains the latest version of the evaluation plan, with key references to existing comparisons of methods.

Study Group 12 would like to invite Study Group 11 to schedule co-located meetings in Geneva, November 2018, and to participate in the SG12 workshop adjacent to the meeting dates, on November 26, 2018. We would like to use the workshop to share our experience, to make the work of overseeing ITU-T SG 12’s mandate and our coordination roles more efficient and effective. 

Attachments:
TD619- Current text of Annex A and Y.1540 clauses.

URL to the Workshop: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.itu.int_en_ITU-2DT_Workshops-2Dand-2DSeminars_qos_201811_Pages_default.aspx&d=DwIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=VX3GNCspVy25DZ0Zvm0i0HhrSCNzJBOHkhMrvjqcqlw&s=SC-YHNEEIUdCW4a7nW3HZ9w6IRJdh2lzJy9W76DJfpo&e=
Attachments:

    oLS59att
    https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2018-2D10-2D26-2Ditu-2Dt-2Dsg-2D12-2Dietf-2Dlsr-2Don-2Dcurrent-2Dstatus-2Dof-2Dthe-2Ddraft-2Drecommendation-2Ditu-2Dt-2Dq3961-2Dreply-2Dto-2Dsg11-2Dls52-2Dattachment-2D1.docx&d=DwIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=VX3GNCspVy25DZ0Zvm0i0HhrSCNzJBOHkhMrvjqcqlw&s=EECf6rVuVJUT-vua7FSXSNXcbwzq2J9XE2BSGzZlFzM&e=