Re: [ippm] Fwd: How should capacity measurement interact with shaping?

[<[e-mail]>]

Hi Joachim,

the QoE measurement campaigns I mentioned were performed on both, laboratory and public LTE networks. The researchers wanted to measure results which relate to consumer experience. I regard QoE measurements as a source giving me general guidance on what's done on a subject also related to QoS, but don't expect them to serve as a blueprint for QoS measurements.  

Pre-loading may be a useful approach to reach "stable" mobile network conditions. Hopefully, there are choices to specify "stable" in a way allowing separate, but compatible implementations of a metric.

My closing statement below  should express, that a metric determining a stable BTC should consider packet loss and queue build up (i.e., increasing one-way latency during "stable" conditions) as indicators of congestion. The highest sending rate not causing congestion seems to be a metric useful for a BTC metric to me. Another one might be a burst size which can be transmitted causing congestion by a queue build up, but no packet drop.

I fully agree that receiver based measurements or receiver perspective, respectively, should be taken into account.

Regards, Ruediger
 

-----Ursprüngliche Nachricht-----
Von: Joachim Fabini <[e-mail]> 
Gesendet: Mittwoch, 25. September 2019 11:55
An: Geib, Rüdiger <[e-mail]>
Cc: [e-mail]
Betreff: Re: AW: [ippm] Fwd: How should capacity measurement interact with shaping?

Hi Ruediger,

thank you for your comments, answers inline.

On 24/09/2019 12:14, [e-mail] wrote:
> Hi Joachim,
> 
> thanks for adding a discussion related to mobile access behavior. I've read some "streaming QoE over LTE" related publications. These are not BTC related, but LTE and CCA interaction also impacts the packet and flow related measurements performed. One conclusion of these publications is, that throughput fluctuations characterize an LTE access network better, than just average throughput values.

Did these publications use isolated LTE networks or public ones?
Whenever (access) network capacity is shared between users (which is the case for most cellular users in a cell) this is a potential source of non-determinism in terms of measurements that is {hard|impossible} to control. Measurement results depend to a substantial extent on the traffic profile and timing of competing devices and users.

> 
> Your discussion whether to consider an access showing 94.5 Mbit/s throughput for a short time and 83 Mbit/s for a longer interval may indicate, that singleton, short interval BTC measurements used as samples to build a (or some) BTC statistics_to_be_defined might be useful approach.  

Yes, I like your proposal. A BTC singleton could be measured after detecting stable conditions and for an interval that is sufficiently short to increase the likelihood of almost constant parameters. The need to reflect the "true" path BTC (i.e., the *receiver's* perspective) and detect/eliminate intermediate node buffering is challenging but imo crucial for such experiments.

> The above QoE experts use receiver statistics, as you suggest too, however their activities are passive, whereas BTC measurements discussed here are active. I add this to indicate, that a mobile access IPPM BTC metric discussion will have to produce own approaches.

The good news about BTC in mobile networks is that (at least in the past, when I did measurements) the scheduling/capacity algorithm predictability and determinism increased for greedy ("the more you ask for, the more you get") users. And BTC measurements are greedy by definition.

> I agree that defining "long term average data rate" in a way which is useful to an implementors use case is desirable (the same holds for guidance on a short term singleton BTC metric).

Yes, but again: in white box measurements (without exact knowledge on the measurement path implementation and configuration) imo there is an imperative need for the *receiver's* perspective. The *sender* itself can't differentiate between stateful policing, shaping and buffering that may happen on the network path.

> To start a discussion: a BTC singleton could consist of back-to-back packets of a length corresponding to an important or expected bottleneck burst tolerance (and/or buffer depth). 

Hmmh. If buffering is known to happen on the path, I consider the BTC singleton that you mention to be a *prerequisite* for the actual measurement BTC singleton: The active measurement must send sufficient data to fill all buffers within the network path under test. Only subsequent packets allow inference on the path's true BTC.

As a side note: an identical approach ("preload") is used for active delay measurements in mobile networks: transferring a large data burst over an idle 3G/4G link makes the radio scheduler to allocate substantial radio resources (i.e. high capacity) to this link. Sending a delay measurement packet immediately after the data burst yields delay values that are by orders of magnitude lower compared to sending the same packets on an almost idle link.

I mentioned this detail because of two aspects:
1. The same mechanism could be used to create adequate measurement prerequisites for mobile cellular networks and for fixed access networks, for BTC and for delay measurements. The packets serve distinct purpose - force capacity allocation in cellular networks vs. filling buffers in fixed network paths - but the same method may work for both cases.
2. Measurement results nowadays depend on the use case (i.e., the context in which they have been acquired). They reflect the performance of the network in a particular state that we can (sometimes) artificially enforce as measurement prerequisite. If the state is the one encountered in typical operation, then the results are representative (which will be typically true for BTC). Otherwise they may be misleading.

> These singletons should be spaced in a way that a loss and may be queue free long term BTC results. 

Sorry, I don't get this last sentence (something missing?).

Regards
Joachim




> -----Ursprüngliche Nachricht-----
> Von: Joachim Fabini <[e-mail]>
> Gesendet: Dienstag, 24. September 2019 11:20
> An: Matt Mathis <mattmathis=[e-mail]>; ALFRED C 
> (AL) <[e-mail]>; Geib, Rüdiger <[e-mail]>
> Cc: [e-mail]
> Betreff: Re: [ippm] Fwd: How should capacity measurement interact with shaping?
> 
> Matt,
> 
> access technology was the first question I wanted to ask yesterday, you answered it in your last email.
> Some more comments, questions, and observations:
> - On-demand capacity allocation over time was/is common for mobile cellular access technologies. Superposition of algorithms at various layers (cellular access capacity management, buffering, users in the cell and CCA) generates challenging traces.
> - Some details on the measurement architecture could help: where did you measure BTC: at the receiver, at the sender, at intermediate nodes? Time granularity/resolution?
> - Al made several good points in his replies; the overshooting of BBR in Fig.5a of http://web.cs.wpi.edu/~claypool/papers/driving-bbr/paper-final.pdf is a hint that *if* there are buffers on the path, BBR will attempt to use them. It's the sequence cache-memory-network: first you fill up all local buffers, then the ones at intermediate nodes on the path, and only later on you can measure the network capacity. Still, 4 seconds for the initial 94.5 Mbit/s burst means truly large buffers...
> 
> You asked for an answer and justification - here we go:
> Boiling down to the essence, your question (which capacity to prefer,
> 94.5 or 83Mb/s) imo addresses a weakness of RFC3148. In particular a more accurate definition and refinement of "long term" could help.
> Quoting RFC 3148: "The intuitive definition of BTC is the expected 
> *long
> term* average data rate (bits per second) of a single ideal TCP implementation over the path in question". Is *long term* 4 seconds?
> 10s? 1000s?
> 
> My interpretation is that you meant a "time interval sufficiently large for CCA to converge and reach a steady state" (D1). However, this definition considers E2E control/CCA *only* and falls short of the pitfall that Al and I attempted to address in RFC 7312 for delay measurements in mobile network measurements: today's (access) networks are stateful (the copper wire abstraction fails). This stateful network policing interacts with the end-to-end CCA and originates artificial, non-representative and non-repeatable measurement results. In DOCSIS (that I did not measure) it seems that buffering is the main issue.
> 
> Some more reasoning on your BTC measurements:
> - In your measurements, when BTC is measured at the sender, the informal definition (D1) seems to hold true for a flow that lasts for slightly less than 4 seconds. But do BTC measurements at the receiver's end yield the same results? They may or may not.
> - If the flow duration exceeds 4 seconds, the convergence criterion (as seen by the sender) is no longer fulfilled: conditions change.
> - If I'm asked if 94.5Mb/s or 83Mb/s is "the" path BTC, my decision depends on the receiver view: (a) if some buffers on the network path fill up, but the data rate at the receiver is only 83Mb/s, then the network path's BTC is 83Mb/s. If there's a shaper on the path that grants an initial "data credit" to the sender at full speed and only later on shapes, i.e, the receiver can measure 94.5Mb/s for the first 4 seconds, then I'd agree to a BTC of 94.5Mb/s. However, only for the first 4 second interval or for the first y Mbytes of data.
> 
> My point is that (unfortunately) the measurement of BTC for a network path must be tied to use cases to be meaningful. I'm truly convinced that with today's networks it's an illusion to have one single, representative value "BTC is x Mb/s". Decoupling BTC from the network path's state and its evolution with time may not be feasible.
> 
> Reasoning on objective BTC:
> First, the measurement of stateful network BTC mandate to take into consideration parameters like (a) the network path implementation, configuration and behavior, (b) the measurement point (sender, receiver, intermediate point), and (c) the measurement flow characteristics (duration, total amount of data? Any parameter that influences on the network state wrt the flow?). These are just examples, it may make sense to structure and categorize these influencing parameters.
> 
> Second, the definition (D1) in RFC3148 could be refined. The *long term* should be made explicit, covering two mandatory components: (a) CCA (end-to-end control) reaching a steady state and (b) the network path under observation reaching a steady state over (c) a well-specified interval.
> 
> In order to be meaningful (and scientifically sound), BTC as an intuitive metric should imho reflect a network path's ability to offer a sustained transfer capacity under (almost) constant conditions for a specified time interval (and/or for a specified amount of data?). Still, architecture and implementation of today's (access) networks make it challenging to identify those intervals during which we have constant conditions.
> 
> So the key question is if one can provide a BTC definition that is accurate, representative and simple/usable for the broad mass. (*) Al and his co-authors have included some key aspect in the capacity-metric draft, some aspects of this thread may be added there, too. But BTC imo is different and even more complex to measure.
> 
> regards Joachim
> PS: (*) Leaving aside uncertainty that we will never be able to handle in measurements because of the capacity-sharing in today's access networks (users-in-a-cell, cell usage).
> 
> 
> 
> On 19/09/2019 23:17, Matt Mathis wrote:
>> Ok, moving the thread to IPPM
>>
>> Some background, we (Measurement Lab) are testing a new transport
>> (TCP) performance measurement tool, based on BBR-TCP.   I'm not ready 
>> to talk about results yet (well ok, it looks pretty good).    (BTW 
>> the BBR algorithm just happens to resemble the algorithm described in
>> draft-morton-ippm-capcity-metric-method-00.)
>>
>> Anyhow we noticed some interesting performance features for number of 
>> ISPs in the US and Europe and I wanted to get some input for how 
>> these cases should be treated.
>>
>> One data point, a single trace saw ~94.5 Mbit/s for ~4 seconds, 
>> fluctuating performance ~75 Mb/s for ~1 second and then stable 
>> performance at ~83Mb/s for the rest of the 10 second test.    If I 
>> were to guess this is probably a policer (shaper?) with a 1 MB token 
>> bucket and a ~83Mb/s token rate (these numbers are not corrected for 
>> header overheads, which actually matter with this tool).  What is 
>> weird about it is that different ingress interfaces to the ISP (peers 
>> or serving locations) exhibit different parameters.
>>
>> Now the IPPM measurement question:   Is the bulk transport capacity 
>> of this link ~94.5 Mbit/s or ~83Mb/s?   Justify your answer....?
>>
>> Thanks,
>> --MM--
>> The best way to predict the future is to create it.  - Alan Kay
>>
>> We must not tolerate intolerance;
>>        however our response must be carefully measured:
>>             too strong would be hypocritical and risks spiraling out 
>> of control;
>>             too weak risks being mistaken for tacit approval.
>>
>> Forwarded Conversation
>> Subject: How should capacity measurement interact with shaping?
>> ------------------------
>>
>> From: *Matt Mathis* <[e-mail] 
>> <mailto:[e-mail]>>
>> Date: Thu, Aug 15, 2019 at 8:55 AM
>> To: MORTON, ALFRED C (AL) <[e-mail] 
>> <mailto:[e-mail]>>
>>
>>
>> We are seeing shapers  with huge bucket sizes, perhaps as larger or 
>> larger than 100 MB.
>>
>> These are prohibitive to test by default, but can have a huge impact 
>> in some common situations.  E.g. downloading software updates.
>>
>> An unconditional pass is not good, because some buckets are small. 
>> What counts as large enough to be ok, and what "derating" is ok?
>>
>> Thanks,
>> --MM--
>> The best way to predict the future is to create it.  - Alan Kay
>>
>> We must not tolerate intolerance;
>>        however our response must be carefully measured:
>>             too strong would be hypocritical and risks spiraling out 
>> of control;
>>             too weak risks being mistaken for tacit approval.
>>
>>
>> ----------
>> From: *MORTON, ALFRED C (AL)* <[e-mail] 
>> <mailto:[e-mail]>>
>> Date: Mon, Aug 19, 2019 at 5:08 AM
>> To: Matt Mathis <[e-mail] 
>> <mailto:[e-mail]>>
>> Cc: CIAVATTONE, LEN <[e-mail] <mailto:[e-mail]>>, 
>> [e-mail] <mailto:[e-mail]> 
>> <[e-mail] <mailto:[e-mail]>>
>>
>>
>> Hi Matt, currently cruising between Crete and Malta,____
>>
>> with about 7 days of vacation remaining – Adding my friend Len.____
>>
>> You know Rüdiger. It appears I’ve forgotten how to typs in 2 
>> weeks____
>>
>> given the number of typos I’ve fixed so far...____
>>
>> __ __
>>
>> We’ve seen big buffers on a basic DOCSIS cable service (downlink >2 
>> sec)____
>>
>> but,____
>>
>>   we have 1-way delay variation or RTT variation limits____
>>
>>   when searching for the max rate, that don’t many packets ____
>>
>>   queue in the buffer____
>>
>> __ __
>>
>>   we want the status messages that result in rate adjustment to 
>> return____
>>
>>  in a reasonable amount of time (50ms + RTT)____
>>
>> __ __
>>
>>   we usually search for 10 seconds, but if we go back and test with 
>> ____
>>
>>   a fixed rate, we can see the buffer growing if the rate is too 
>> high.____
>>
>> __ __
>>
>>   There will eventually be a discussion on the thresholds we use____
>>
>>   in the search // load rate control algorithm. The copy of ____
>>
>>   Y.1540 I sent you has a simple one, we moved beyond that now____
>>
>>   (see the slides I didn’t get to present at IETF).____
>>
>> __ __
>>
>>   There is value in having some of this discussion on IPPM-list,____
>>
>>   so we get some **agenda time at IETF-106** ____
>>
>> __ __
>>
>> We measure rate and performance, with some performance limits____
>>
>> built-in.  Pass/Fail is another step, de-rating too (made sense ____
>>
>> with MBM “target_rate”).____
>>
>> __ __
>>
>> Al____
>>
>>
>>
>> ----------
>> From: <[e-mail] <mailto:[e-mail]>>
>> Date: Mon, Aug 26, 2019 at 12:05 AM
>> To: <[e-mail] <mailto:[e-mail]>>
>> Cc: <[e-mail] <mailto:[e-mail]>>, <[e-mail] 
>> <mailto:[e-mail]>>
>>
>>
>> Hi Al,____
>>
>> __ __
>>
>> thanks for keeping me involved. I don’t have a precise answer and 
>> doubt, that there will be a single universal truth. ____
>>
>> __ __
>>
>> If the aim is only to determine the IP bandwidth of an access, then 
>> we aren’t interested in filling a buffer. Buffering events may occur, 
>> some of which are useful and to be expected, whereas others are not 
>> desired:____
>>
>> __ __
>>
>>   * Sender shaping behavior may matter (is traffic at the source CBR or
>>     is it bursty)____
>>   * Random collisions should be tolerated at the access whose bandwidth
>>     is to be measured.____
>>   * Limiting packet drop due to buffer overflow is a design aim or an
>>     important part of the algorithm, I think.____
>>   * Shared media might create bursts. I’m not an expert in the area, but
>>     there’s an “is bandwidth available” check in some cases between a
>>     central sender using a shared medium and the receivers connected.
>>     WiFi and may be other wireless equipment buffers packets also to
>>     optimize wireless resource optimization.____
>>   * It might be an idea to mark some flows by ECN, once there’s a guess
>>     on a sending bitrate when to expect no or very little packet drop.
>>     Today, this is experimental. CE marks by an ECN capable device
>>     should be expected roughly once queuing starts.____
>>
>> __ __
>>
>> Practically, the set-up should be configurable with commodity hard- 
>> and software and all metrics should be measurable at the receiver.
>> Burstiness of traffic and a distinction between queuing events which 
>> are to be expected and (undesired) queue build up are the events to 
>> be distinguished. I hope that can be done with commodity hard- and 
>> software. I at least am not able to write down a simple metric 
>> distinguishing queues to be expected from (undesired) queue build up 
>> causing congestion. The hard- and software to be used should be part 
>> of the solution, not part of the problem (bursty source traffic and 
>> timestamps with insufficient accuracy to detect queues are what I’d 
>> like to avoid).____
>>
>> __ __
>>
>> I’d suggest to move discussion to the list. ____
>>
>> __ __
>>
>> Regards,____
>>
>> __ __
>>
>> Rüdiger____
>>
>>
>>
>> ----------
>> From: *MORTON, ALFRED C (AL)* <[e-mail] 
>> <mailto:[e-mail]>>
>> Date: Thu, Sep 19, 2019 at 7:01 AM
>> To: Ruediger.Geib@telekom..de <mailto:[e-mail]> 
>> <[e-mail] <mailto:[e-mail]>>
>> Cc: CIAVATTONE, LEN <[e-mail] <mailto:[e-mail]>>, 
>> [e-mail] <mailto:[e-mail]> 
>> <[e-mail] <mailto:[e-mail]>>
>>
>>
>> I’m catching-up with this thread again, but before I reply:____
>>
>> __ __
>>
>> *** Any objection to moving this discussion to IPPM-list ?? ***____
>>
>> __ __
>>
>> @Matt – this is a question to you at this point...____
>>
>> __ __
>>
>> thanks,____
>>
>> Al____
>>
>> __ __
>>
>> *From:*[e-mail] <mailto:[e-mail]> 
>> [mailto:[e-mail] <mailto:[e-mail]>]
>> *Sent:* Monday, August 26, 2019 3:05 AM
>> *To:* MORTON, ALFRED C (AL) <[e-mail] 
>> <mailto:[e-mail]>>
>> *Cc:* CIAVATTONE, LEN <[e-mail] <mailto:[e-mail]>>; 
>> [e-mail] <mailto:[e-mail]>
>> *Subject:* AW: How should capacity measurement interact with 
>> shaping?____
>>
>> __ __
>>
>> Hi Al,____
>>
>> __ __
>>
>> thanks for keeping me involved. I don’t have a precise answer and 
>> doubt, that there will be a single universal truth. ____
>>
>> __ __
>>
>> If the aim is only to determine the IP bandwidth of an access, then 
>> we aren’t interested in filling a buffer. Buffering events may occur, 
>> some of which are useful and to be expected, whereas others are not 
>> desired:____
>>
>> __ __
>>
>> __-        __Sender shaping behavior may matter (is traffic at the 
>> source CBR or is it bursty)____
>>
>> __-        __Random collisions should be tolerated at the access 
>> whose bandwidth is to be measured.____
>>
>> __-        __Limiting packet drop due to buffer overflow is a design 
>> aim or an important part of the algorithm, I think.____
>>
>> __-        __Shared media might create bursts. I’m not an expert in 
>> the area, but there’s an “is bandwidth available” check in some cases 
>> between a central sender using a shared medium and the receivers 
>> connected. WiFi and may be other wireless equipment buffers packets 
>> also to optimize wireless resource optimization.____
>>
>> __-        __It might be an idea to mark some flows by ECN, once 
>> there’s a guess on a sending bitrate when to expect no or very little 
>> packet drop. Today, this is experimental. CE marks by an ECN capable 
>> device should be expected roughly once queuing starts.____
>>
>>
>>
>>
>> _______________________________________________
>> ippm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ippm
>>
> 
>