Re: [ippm] September Summary on Max IP-Layer Capacity Metric

"MORTON, ALFRED C (AL)" <acm@research.att.com> Mon, 07 October 2019 16:14 UTC

Return-Path: <acm@research.att.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4DF01120041 for <ippm@ietfa.amsl.com>; Mon, 7 Oct 2019 09:14:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.601
X-Spam-Level:
X-Spam-Status: No, score=-2.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UmVFjGvZ2Ltc for <ippm@ietfa.amsl.com>; Mon, 7 Oct 2019 09:14:14 -0700 (PDT)
Received: from mx0a-00191d01.pphosted.com (mx0a-00191d01.pphosted.com [67.231.149.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A58CE120043 for <ippm@ietf.org>; Mon, 7 Oct 2019 09:14:14 -0700 (PDT)
Received: from pps.filterd (m0049297.ppops.net [127.0.0.1]) by m0049297.ppops.net-00191d01. (8.16.0.42/8.16.0.42) with SMTP id x97G3bYb019179 for <ippm@ietf.org>; Mon, 7 Oct 2019 12:14:13 -0400
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com [144.160.112.28]) by m0049297.ppops.net-00191d01. with ESMTP id 2vg7hf9sm8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for <ippm@ietf.org>; Mon, 07 Oct 2019 12:14:03 -0400
Received: from enaf.dadc.sbc.com (localhost [127.0.0.1]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id x97GCBaA061529 for <ippm@ietf.org>; Mon, 7 Oct 2019 11:12:12 -0500
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [135.46.181.156]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id x97GC4E1061208 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for <ippm@ietf.org>; Mon, 7 Oct 2019 11:12:05 -0500
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [127.0.0.1]) by zlp30497.vci.att.com (Service) with ESMTP id 24BB84009E80 for <ippm@ietf.org>; Mon, 7 Oct 2019 16:12:04 +0000 (GMT)
Received: from clpi183.sldc.sbc.com (unknown [135.41.1.46]) by zlp30497.vci.att.com (Service) with ESMTP id F2EE44009E81 for <ippm@ietf.org>; Mon, 7 Oct 2019 16:12:03 +0000 (GMT)
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id x97GC3r2025120 for <ippm@ietf.org>; Mon, 7 Oct 2019 11:12:03 -0500
Received: from mail-green.research.att.com (mail-green.research.att.com [135.207.255.15]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id x97GBv9p024726 for <ippm@ietf.org>; Mon, 7 Oct 2019 11:11:58 -0500
Received: from exchange.research.att.com (njbdcas1.research.att.com [135.197.255.61]) by mail-green.research.att.com (Postfix) with ESMTP id 20EDFE1C22 for <ippm@ietf.org>; Mon, 7 Oct 2019 12:09:01 -0400 (EDT)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0468.000; Mon, 7 Oct 2019 12:11:47 -0400
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: "ippm@ietf.org" <ippm@ietf.org>
Thread-Topic: September Summary on Max IP-Layer Capacity Metric
Thread-Index: AdV3Dk0U0CN8YEE8RCWwGfOaryjfnQGAJW0Q
Date: Mon, 07 Oct 2019 16:11:46 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CFA0AFEE95@njmtexg5.research.att.com>
References: <4D7F4AD313D3FC43A053B309F97543CFA0AFBAA6@njmtexg5.research.att.com>
In-Reply-To: <4D7F4AD313D3FC43A053B309F97543CFA0AFBAA6@njmtexg5.research.att.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [69.141.203.172]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-10-07_03:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910070153
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/VoVCL2IHsHbIHQwDTtLi-LR354A>
Subject: Re: [ippm] September Summary on Max IP-Layer Capacity Metric
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 07 Oct 2019 16:14:17 -0000

IPPM:

In a side discussion on a related topic, which the authors of
https://tools.ietf.org/html/draft-morton-ippm-capcity-metric-method-00
will share on the ippm-list shortly, Rüdiger reminded me that Joachim
asked Matt the question of whether BBR Capacity is measured at 
sender or receiver or both (the question applies to both 
metrics: Max IP-layer and Bulk Transfer Capacity). 

Joachim wrote:
...
> Some more reasoning on your BTC measurements:
> - In your measurements, when BTC is measured at the sender, the informal
> definition (D1) seems to hold true for a flow that lasts for slightly
> less than 4 seconds. But do BTC measurements at the receiver's end yield
> the same results? They may or may not.
> - If the flow duration exceeds 4 seconds, the convergence criterion (as
> seen by the sender) is no longer fulfilled: conditions change.
> - If I'm asked if 94.5Mb/s or 83Mb/s is "the" path BTC, my decision
> depends on the receiver view: (a) if some buffers on the network path
> fill up, but the data rate at the receiver is only 83Mb/s, then the
> network path's BTC is 83Mb/s. If there's a shaper on the path that
> grants an initial "data credit" to the sender at full speed and only
> later on shapes, i.e, the receiver can measure 94.5Mb/s for the first 4
> seconds, then I'd agree to a BTC of 94.5Mb/s. However, only for the
> first 4 second interval or for the first y Mbytes of data.
>
...

Sender or Receiver view was a question I first grappled-with during the 
lab testing with shaper-based "ground-truth" we conducted before 
approaching IPPM with this proposal [ref: many Liaisons from SG12].

We have concluded that *both* are needed, but we omitted the 
Sender Rate Metric from the draft.  It's actually very useful
to check that the Sender achieved the desired bit rate, and to 
know when it doesn't in practice! 

So, we add one more item to address in the draft:

@@@@ Add a metric on Sender Rate, as both a
  + Parameter to the IP-layer Capacity Metric Definition
  + A Metric at the Src, partly as a check that the desired 
    Parameter was achieved, or was capable of being achieved.

Thanks for this point, Joachim & Rüdiger. 
It was a clear omission in the draft, 
and should be an easy fix because we have 
provided the definition in other work/SDOs.

Al

PS: We have both in Lab Benchmarking, where RFC 2544 Throughput is
based on Offered Load, and RFC 2889 Max Frame Rate is defined 
at the receiver. The useful cross-over between BMWG & IPPM continues. 

> -----Original Message-----
> From: ippm [mailto:ippm-bounces@ietf.org] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Sunday, September 29, 2019 5:41 PM
> To: ippm@ietf.org
> Subject: [ippm] September Summary on Max IP-Layer Capacity Metric
> 
> 
> IPPM List September Summary on Max IP-Layer Capacity Metric
> (Re: [ippm] How should capacity measurement interact with shaping?)
> currently draft-morton-ippm-capcity-metric-measurement-00
> 
> We've had a very good discussion of many important
> aspects of IP layer Capacity Metric/Measurements, including:
> 
> + Recognizing how an alt. flow control for TCP (BBR) uses a similar metric
> + Reporting the results under unusual circumstances
> + Bringing IPPM's documented experience and literature to the problem
> + Gaining experience from each-other's measurements/research
> + Suggestion of related work areas
> 
> It's useful to summarize many pages of discussion from time to
> time: we can capture (what the summarizer thinks) we learned,
> and new readers can join the discussion more easily.
> With those goals in mind, a humble attempt to summarize follows.
> Feel free to set me straight in a concise way, of course.
> 
> @@@@ is a flag for take-aways; items to address in the draft.
> 
> Matt Mathis engaged the "capcity" draft authors shortly
> after IETF-105, and kindly agreed to foster wider review
> on the ippm-list. There's a whole lot of *shaping* going on [0].
> Matt's M-Lab measurements revealed a clear case of bi-modal
> maximum rates (94 & 83 Mbps), consistent with a service feature
> in the context of Shaping, and Rüdiger shared his experiences
> with fixed access shaper design.
> @@@@ A clear take-away is that reporting must account for such a
> bimodal feature, if/when measured.
> @@@@ Also, that wide-spread measurements will encounter wide-spread
> behaviors - testing should continue + expect some evolution.
> 
> Joachim and Rüdiger discussed the situation further, confirming
> how buffers play a big part in the assessment and performance.
> When answering the reporting question, the measurement time interval
> (long-term?, many different shapers and on-demand technology
> may be encountered, as anticipated in RFC 7312) play a key role.
> Joachim also provided two key points of reasoning for BTC (RFC 3148):
> categorize the influencing factors and refine the 3148 definition.
> The discussion covered LTE public networks with on-demand access
> and shared resources.
> 
> @@@@ IMO, many of the above challenges fall on the measurement
> methodology: allow for traffic & time to initiate an on-demand access.
> @@@@ Also, results depend on the sending stream characteristics;
> we've known this for a long time, still need to keep it front of mind.
> @@@@ Max IP-Layer Capacity and RFC 3148 BTC (goodput) are different
> metrics. Max IP-layer Capacity is like the theoretical goal for goodput.
> 
> @@@@ This is a big one: when the path we measure is state-full based on
> many factors, the Parameter "Time of day" when a test starts is not
> enough info. We need to know the time from the beginning of a
> measured flow, and how the flow is constructed including how much
> traffic has already been sent on that flow, because state-change
> may be based on time or bytes sent or both. Re-read RFC 7312.
> 
> @@@@ The Singleton and Statistic formulations of IPPM's framework
> RFC 2330 are still valuable in this context, possibly combined with
> results criteria ("stable" for X singletons, non-arbitrary threshold
> needed to define "stable").
> 
> Rüdiger proposed a back-to-back stream for BTC characterization.
> Joachim felt this b2b test might be a pre-requisite to measure a
> BTC singleton.
> [acm] it's a tricky test in production networks, see [1]
> 
> @@@@ Measurements depend on the access network and the use case.
> Here, the use case is to assess the maximum capacity of the
> access network, with specific performance criteria used in the
> measurement.
> 
> Finally, an exchange between Ignacio and Rüdiger brings us
> back to first-principles: What are you trying to measure, and
> what does it mean? What does it matter to demonstrate that
> a portion of the network can reach a published value?
> What capacity is available 100% of the time: you cannot
> make measurements that saturate the network 100% of the time?
> Rüdiger responded that this effort has very specific goals,
> to demonstrate that the performance promised is present when
> requested to do so, consistent with the metric proposed.
> There are *many* other metrics, such as available BW.
> Ignacio had some measurement proposals for what may be a
> different network performance metric (IMO).
> 
> @@@@ Goals made clearer in the next draft, if possible.
> 
> Well, that's a long summary, and we have identified many work
> items for the draft. We also have more measurements (and
> therefore, more useful experiences) coming.
> 
> Thanks to all who commented so far, very helpful stuff.
> We look forward to additional discussion and suggestions! [2]
> 
> regards,
> Al
> 
> [0] apologies to Jerry Lee Louis:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.youtube.com_watch-3Fv-3D1dC0DseCyYE&d=DwIFAw&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=bbgCkEjNrPRLEewNG6ZmB_sgyglVu
> M-SdbxPtJaxIWQ&s=neeGM557r0t9U2sr1X6A7GClYDTLjgvE04-cMFxL5MA&e=
> 
> [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Db2b-2Dframe-
> 2D00&d=DwIFAw&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=bbgCkEjNrPRLEewNG6ZmB_sgyglVu
> M-SdbxPtJaxIWQ&s=jqU4ecqKIViAJthqNnzDl7B2eHGmjAndjVhLw4YsP8Y&e=
> 
> [2] It would be good to create threads on specific topics in future, but
> Keep those cards and letters coming-in, folks!
> 
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_ippm&d=DwIFAw&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=bbgCkEjNrPRLEewNG6ZmB_sgyglVu
> M-SdbxPtJaxIWQ&s=KLFtWoMazukYq_Aqq2C67G4rzNW5De7fMNKdbYq9smQ&e=