[ippm] September Summary on Max IP-Layer Capacity Metric

["MORTON, ALFRED C (AL)" <[e-mail]>]

IPPM List September Summary on Max IP-Layer Capacity Metric
(Re: [ippm] How should capacity measurement interact with shaping?)
currently draft-morton-ippm-capcity-metric-measurement-00

We've had a very good discussion of many important
aspects of IP layer Capacity Metric/Measurements, including:

+ Recognizing how an alt. flow control for TCP (BBR) uses a similar metric 
+ Reporting the results under unusual circumstances
+ Bringing IPPM's documented experience and literature to the problem
+ Gaining experience from each-other's measurements/research
+ Suggestion of related work areas

It's useful to summarize many pages of discussion from time to
time: we can capture (what the summarizer thinks) we learned, 
and new readers can join the discussion more easily. 
With those goals in mind, a humble attempt to summarize follows.
Feel free to set me straight in a concise way, of course.

@@@@ is a flag for take-aways; items to address in the draft.

Matt Mathis engaged the "capcity" draft authors shortly 
after IETF-105, and kindly agreed to foster wider review
on the ippm-list. There's a whole lot of *shaping* going on [0].
Matt's M-Lab measurements revealed a clear case of bi-modal
maximum rates (94 & 83 Mbps), consistent with a service feature 
in the context of Shaping, and Rüdiger shared his experiences 
with fixed access shaper design.
@@@@ A clear take-away is that reporting must account for such a
bimodal feature, if/when measured. 
@@@@ Also, that wide-spread measurements will encounter wide-spread
behaviors - testing should continue + expect some evolution.

Joachim and Rüdiger discussed the situation further, confirming
how buffers play a big part in the assessment and performance.
When answering the reporting question, the measurement time interval
(long-term?, many different shapers and on-demand technology
may be encountered, as anticipated in RFC 7312) play a key role.
Joachim also provided two key points of reasoning for BTC (RFC 3148):
categorize the influencing factors and refine the 3148 definition.
The discussion covered LTE public networks with on-demand access 
and shared resources. 

@@@@ IMO, many of the above challenges fall on the measurement
methodology: allow for traffic & time to initiate an on-demand access.
@@@@ Also, results depend on the sending stream characteristics;
we've known this for a long time, still need to keep it front of mind.
@@@@ Max IP-Layer Capacity and RFC 3148 BTC (goodput) are different
metrics. Max IP-layer Capacity is like the theoretical goal for goodput.

@@@@ This is a big one: when the path we measure is state-full based on
many factors, the Parameter "Time of day" when a test starts is not 
enough info. We need to know the time from the beginning of a 
measured flow, and how the flow is constructed including how much 
traffic has already been sent on that flow, because state-change
may be based on time or bytes sent or both. Re-read RFC 7312.

@@@@ The Singleton and Statistic formulations of IPPM's framework
RFC 2330 are still valuable in this context, possibly combined with
results criteria ("stable" for X singletons, non-arbitrary threshold
needed to define "stable"). 

Rüdiger proposed a back-to-back stream for BTC characterization.
Joachim felt this b2b test might be a pre-requisite to measure a
BTC singleton. 
[acm] it's a tricky test in production networks, see [1]

@@@@ Measurements depend on the access network and the use case.
Here, the use case is to assess the maximum capacity of the 
access network, with specific performance criteria used in the 
measurement.

Finally, an exchange between Ignacio and Rüdiger brings us 
back to first-principles: What are you trying to measure, and
what does it mean? What does it matter to demonstrate that 
a portion of the network can reach a published value?
What capacity is available 100% of the time: you cannot 
make measurements that saturate the network 100% of the time? 
Rüdiger responded that this effort has very specific goals,
to demonstrate that the performance promised is present when
requested to do so, consistent with the metric proposed.
There are *many* other metrics, such as available BW.
Ignacio had some measurement proposals for what may be a 
different network performance metric (IMO).

@@@@ Goals made clearer in the next draft, if possible.

Well, that's a long summary, and we have identified many work
items for the draft. We also have more measurements (and 
therefore, more useful experiences) coming. 

Thanks to all who commented so far, very helpful stuff.
We look forward to additional discussion and suggestions! [2]

regards, 
Al

[0] apologies to Jerry Lee Louis:  https://www.youtube.com/watch?v=1dC0DseCyYE

[1] https://tools.ietf.org/html/draft-ietf-bmwg-b2b-frame-00

[2] It would be good to create threads on specific topics in future, but
Keep those cards and letters coming-in, folks!