Re: [tsvwg] draft minutes from interim

[Sebastian Moeller <[e-mail]>]

Hi All,

Summary: Focussing only on raw time courses or aggregate statistics is not going to cut it, looking at both/all levels is required.

more below,


> On Mar 6, 2020, at 18:27, Bob Briscoe <[e-mail]> wrote:
> 
> Wes,
> 
> 
> On 05/03/2020 03:04, Wesley Eddy wrote:
>> Draft minutes from the interim are available at:
>> 
>> https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/materials/minutes-interim-2020-tsvwg-01-202002200900 
>> 
> Thx for these. Look like a pretty good summary.
> 
> For those who might want to find these meeting materials in future without saving this specific URL, interim meetings are linked off the data tracker IETF proceedings page:
> https://www.ietf.org/how/meetings/interim/
> 
>> Please send any corrections or improvements.
> During Jonathan's slides, there was a conversation that seems to be completely missing where Greg asked for SCE scenarios to not just show experiments as selected individual time series that could be "cherry picked", but to usually give results over a range of realistic link rates and RTTs and a range of traffic scenarios. Also there was a request to present summary "figures of merit" e.g. queue delay, utilization, rate ratio, packet drop, ECN marking, normalized flow completion time, etc. Also to show "whiskers", e.g. 99-%ile as well as mean, across a number of experiment runs.


	[SM] I want to add, that comprehensive tests are obviously a good thing, but that by only looking at aggregate statistics and log-scales it is quite easy to miss catastrophic failures like the dualQ fairness breakdown at short RTTs. This result has been there pretty much from the beginning and yet it has not received enough attention, presumably because it was buried in big tomb of data and tests.
	Why do I keep harping on that so much? Because this failure indicates that dualQ can be gamed, any sufficiently unresponsive paced flow will be able to effectively slow everything else down to a trickle, and flows in the non-LL queue will not even be able to "fight back" with an equal amount of unresponsiveness. And IMHO that is considerably worse than the status quo, where equally responsive flows will end up sharing roughly equitable. Insisting that the failure as currently observed only happens in a "corner-case" completely ignores that (in adddition to the fact that that corner-case, short RTT flows is exactly where the CDNd internet has been moving closer and closer to).

	In short both comprehensive presentation of aggregate summary statistics for different cases as well as detailed presentations of time-resolved identified problematic cases seem appropriate.
	Regarding statistical reporting, in box whisker plots, as appropriate for expected non-normal data like delays, the mean is far less relevant than the median... also great idea to define what the whiskers are supposed to represent. The "figures of merit" idea, I believe is not unproblematic, as it tempts to only look and compare aggregate 2nd level statistics which can occlude what is actually happening at the "raw" data level. As always, one needs to also look at the raw time courses if only to avoid falling into common GIGO traps.


> 
> (The conversation started by audio, then continued on the chat). Time series are useful to dig into particular anomalous behaviours, but not really suitable for checking the operating range or to compare solutions over a range of scenarios. Higher percentile figures are essential for queue delay to determine how a real-time app would cope.

	[SM] Let's be clear, CDF type plots for delay are relevant and have merit, but the idea that you seem to be pushing here, that L4S will allow to deliver arbitrary tight real-time demands over the wider best-effort internet needs to be quashed, because it will not. Sure L4S might push down the average queueing delay over a short path down far enough to allow some tighter real-time demands than before... but it will not the end-all of queueing delay and jitter. The moment you replace the traditional bad >100ms queueing delay under load traffic shaper at the end-user/ISP transition all other variances < 100ms will start to matter...


> 
>> 
>> We are planning to get the WebEx recording posted to the IETF YouTube channel shortly, but need to do a little bit more work on that first.
> Thank you.
> And the chat log hopefully?
> 
> Now I've seen some hints at the conversation I couldn't hear during the chairs slides (audio problems), I'll start following-up on the tsvwg list.
> 
> Cheers
> 
> 
> Bob
>> 
>> 
>> 
>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/
>