[ippm] September Summary on Max IP-Layer Capacity Metric

"MORTON, ALFRED C (AL)" <acm@research.att.com> Sun, 29 September 2019 21:41 UTC

Return-Path: <acm@research.att.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 4B963120077 for <ippm@ietfa.amsl.com>; Sun, 29 Sep 2019 14:41:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.1
X-Spam-Status: No, score=0.1 tagged_above=-999 required=5 tests=[BAYES_50=0.8, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id Pqit8us58_pK for <ippm@ietfa.amsl.com>; Sun, 29 Sep 2019 14:41:27 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0a-00191d01.pphosted.com []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6FE48120073 for <ippm@ietf.org>; Sun, 29 Sep 2019 14:41:27 -0700 (PDT)
Received: from pps.filterd (m0053301.ppops.net []) by mx0a-00191d01.pphosted.com ( with SMTP id x8TLZBpL018010 for <ippm@ietf.org>; Sun, 29 Sep 2019 17:41:25 -0400
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com []) by mx0a-00191d01.pphosted.com with ESMTP id 2vb429rn6g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for <ippm@ietf.org>; Sun, 29 Sep 2019 17:41:25 -0400
Received: from enaf.dadc.sbc.com (localhost []) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id x8TLfOYs011631 for <ippm@ietf.org>; Sun, 29 Sep 2019 16:41:24 -0500
Received: from zlp30495.vci.att.com (zlp30495.vci.att.com []) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id x8TLfLp3011578 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for <ippm@ietf.org>; Sun, 29 Sep 2019 16:41:21 -0500
Received: from zlp30495.vci.att.com (zlp30495.vci.att.com []) by zlp30495.vci.att.com (Service) with ESMTP id 166054005C2B for <ippm@ietf.org>; Sun, 29 Sep 2019 21:41:21 +0000 (GMT)
Received: from clpi183.sldc.sbc.com (unknown []) by zlp30495.vci.att.com (Service) with ESMTP id E87CE4005C2A for <ippm@ietf.org>; Sun, 29 Sep 2019 21:41:20 +0000 (GMT)
Received: from sldc.sbc.com (localhost []) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id x8TLfKQp018730 for <ippm@ietf.org>; Sun, 29 Sep 2019 16:41:20 -0500
Received: from mail-blue.research.att.com (mail-blue.research.att.com []) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id x8TLfEFX018403 for <ippm@ietf.org>; Sun, 29 Sep 2019 16:41:15 -0500
Received: from exchange.research.att.com (njbdcas1.research.att.com []) by mail-blue.research.att.com (Postfix) with ESMTP id 8378CF0DEA for <ippm@ietf.org>; Sun, 29 Sep 2019 17:41:14 -0400 (EDT)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0468.000; Sun, 29 Sep 2019 17:41:07 -0400
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: "ippm@ietf.org" <ippm@ietf.org>
Thread-Topic: September Summary on Max IP-Layer Capacity Metric
Thread-Index: AdV3Dk0U0CN8YEE8RCWwGfOaryjfnQ==
Date: Sun, 29 Sep 2019 21:41:06 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CFA0AFBAA6@njmtexg5.research.att.com>
Accept-Language: en-US
Content-Language: en-US
x-originating-ip: []
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-09-29_13:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1909290247
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/Dwufxbd8-gxSyfreaOfWYLA7J_0>
Subject: [ippm] September Summary on Max IP-Layer Capacity Metric
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 29 Sep 2019 21:41:29 -0000

IPPM List September Summary on Max IP-Layer Capacity Metric
(Re: [ippm] How should capacity measurement interact with shaping?)
currently draft-morton-ippm-capcity-metric-measurement-00

We've had a very good discussion of many important
aspects of IP layer Capacity Metric/Measurements, including:

+ Recognizing how an alt. flow control for TCP (BBR) uses a similar metric 
+ Reporting the results under unusual circumstances
+ Bringing IPPM's documented experience and literature to the problem
+ Gaining experience from each-other's measurements/research
+ Suggestion of related work areas

It's useful to summarize many pages of discussion from time to
time: we can capture (what the summarizer thinks) we learned, 
and new readers can join the discussion more easily. 
With those goals in mind, a humble attempt to summarize follows.
Feel free to set me straight in a concise way, of course.

@@@@ is a flag for take-aways; items to address in the draft.

Matt Mathis engaged the "capcity" draft authors shortly 
after IETF-105, and kindly agreed to foster wider review
on the ippm-list. There's a whole lot of *shaping* going on [0].
Matt's M-Lab measurements revealed a clear case of bi-modal
maximum rates (94 & 83 Mbps), consistent with a service feature 
in the context of Shaping, and Rüdiger shared his experiences 
with fixed access shaper design.
@@@@ A clear take-away is that reporting must account for such a
bimodal feature, if/when measured. 
@@@@ Also, that wide-spread measurements will encounter wide-spread
behaviors - testing should continue + expect some evolution.

Joachim and Rüdiger discussed the situation further, confirming
how buffers play a big part in the assessment and performance.
When answering the reporting question, the measurement time interval
(long-term?, many different shapers and on-demand technology
may be encountered, as anticipated in RFC 7312) play a key role.
Joachim also provided two key points of reasoning for BTC (RFC 3148):
categorize the influencing factors and refine the 3148 definition.
The discussion covered LTE public networks with on-demand access 
and shared resources. 

@@@@ IMO, many of the above challenges fall on the measurement
methodology: allow for traffic & time to initiate an on-demand access.
@@@@ Also, results depend on the sending stream characteristics;
we've known this for a long time, still need to keep it front of mind.
@@@@ Max IP-Layer Capacity and RFC 3148 BTC (goodput) are different
metrics. Max IP-layer Capacity is like the theoretical goal for goodput.

@@@@ This is a big one: when the path we measure is state-full based on
many factors, the Parameter "Time of day" when a test starts is not 
enough info. We need to know the time from the beginning of a 
measured flow, and how the flow is constructed including how much 
traffic has already been sent on that flow, because state-change
may be based on time or bytes sent or both. Re-read RFC 7312.

@@@@ The Singleton and Statistic formulations of IPPM's framework
RFC 2330 are still valuable in this context, possibly combined with
results criteria ("stable" for X singletons, non-arbitrary threshold
needed to define "stable"). 

Rüdiger proposed a back-to-back stream for BTC characterization.
Joachim felt this b2b test might be a pre-requisite to measure a
BTC singleton. 
[acm] it's a tricky test in production networks, see [1]

@@@@ Measurements depend on the access network and the use case.
Here, the use case is to assess the maximum capacity of the 
access network, with specific performance criteria used in the 

Finally, an exchange between Ignacio and Rüdiger brings us 
back to first-principles: What are you trying to measure, and
what does it mean? What does it matter to demonstrate that 
a portion of the network can reach a published value?
What capacity is available 100% of the time: you cannot 
make measurements that saturate the network 100% of the time? 
Rüdiger responded that this effort has very specific goals,
to demonstrate that the performance promised is present when
requested to do so, consistent with the metric proposed.
There are *many* other metrics, such as available BW.
Ignacio had some measurement proposals for what may be a 
different network performance metric (IMO).

@@@@ Goals made clearer in the next draft, if possible.

Well, that's a long summary, and we have identified many work
items for the draft. We also have more measurements (and 
therefore, more useful experiences) coming. 

Thanks to all who commented so far, very helpful stuff.
We look forward to additional discussion and suggestions! [2]


[0] apologies to Jerry Lee Louis:  https://www.youtube.com/watch?v=1dC0DseCyYE

[1] https://tools.ietf.org/html/draft-ietf-bmwg-b2b-frame-00

[2] It would be good to create threads on specific topics in future, but
Keep those cards and letters coming-in, folks!