Re: [tsvwg] draft minutes from interim

Sebastian Moeller <moeller0@gmx.de> Wed, 25 March 2020 08:12 UTC

Return-Path: <moeller0@gmx.de>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4D1CE3A0B3E for <tsvwg@ietfa.amsl.com>; Wed, 25 Mar 2020 01:12:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.111
X-Spam-Level:
X-Spam-Status: No, score=-3.111 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_MSPIKE_H2=-1.463, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xqrG7n-OUeKh for <tsvwg@ietfa.amsl.com>; Wed, 25 Mar 2020 01:12:11 -0700 (PDT)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id F1B913A0837 for <tsvwg@ietf.org>; Wed, 25 Mar 2020 01:12:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1585123886; bh=WNpRhFadtvU/KyR5pxVo3cFBGcg8KfymTzn8SEYkEgI=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=le8BrVBvyOAA6o+b27qaveolbm/su5c9gzZ7s7/1kqYaFKJL7HAfRNn4QGChJs27A AXEWzIYMtAlfoKJ9eNyOGFYTr6zdjvfpAd+D0ecOwTO35sqhRCHVSNzDhyO8wKqbTj fbhg3qK1LOlDsesfS6ZyPYC4Rq8x6Wj0mJvsDfoY=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [10.11.12.22] ([134.76.241.253]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MKbkC-1iyOsk1eJQ-00Kyz7; Wed, 25 Mar 2020 09:11:26 +0100
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Sebastian Moeller <moeller0@gmx.de>
In-Reply-To: <46003417-5ff0-b7b2-4038-0285b85a769f@bobbriscoe.net>
Date: Wed, 25 Mar 2020 09:11:24 +0100
Cc: Wesley Eddy <wes@mti-systems.com>, "tsvwg@ietf.org" <tsvwg@ietf.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <ABE27C9F-2AE6-4F55-A70F-3D605AFBF900@gmx.de>
References: <504d5b50-03bd-fd50-b968-b2def7ed6335@mti-systems.com> <46003417-5ff0-b7b2-4038-0285b85a769f@bobbriscoe.net>
To: Bob Briscoe <in@bobbriscoe.net>
X-Mailer: Apple Mail (2.3445.104.11)
X-Provags-ID: V03:K1:a7jRLu3D4SyGRv/Ogs3TW/8Z9J1Rvuz4krhyReRwxnN+HqxR1Ex HMmjX5DDwxIrg16e6zZWvNgqPrJEWVLZUAxls1d7gOEnjoUHT1WZUOMyUJcZDo/kx3TGiyr crLzadMCGIPfwuLOovfhWhAVNUTK7Qyb+/iZMF7Eiud7LIh6mP33wLS2ymHue63/P1mLucF HCxnYjc+zLm87zhMhIyng==
X-UI-Out-Filterresults: notjunk:1;V03:K0:reC/nE/Yi8o=:crRK2L7h3l5o2yJHOMHC4V 2c3uK+0/Td2nf2R6ObkIzDNrSzSUROIwlgGTLlt1cLGyO3z+2FtgvInpqJ0BOIhxllbp7L7+L wVQ79anzmr5yfo2zeBXYQ6UyoRYyOItqp9wScIs4jLotSfACI4PF3HSZtAdNmC+rbYC54EaZj dciYZLxS8vL2hoyMrG0gVLcA/PVtKRm1zxXbpxr94iteTBjHOTPoVH6qJY0OSM2NDemOTCw5/ Qlw0CtS7XA3xU2AmCcEpvRR7FN0aOSfbewLrNLMnuQdFJq5UFVo7nmz0Uhr/V2aVTE3K96oiw dBXxDgmh0U7/ZtuQK1HI+JDiJdDdqcw5y4GSHpUteAgMRQVCgJw7nvp7bar6EmdAEchMX47Q9 rAtZ/mvttE3hm7nyW6Lq28Ov6JVJiftiE7Xdhs0f4cK8wkRq2IXW8oj4p9fuS+/EW0wsSxXhX RgGbjCJ3tDLe7cE0zav7QYusotEK/M+iCEVXcX3TZkgvdKzOhaX13c9hwWmgLBsnND4a0ImVy Ka/EKtlQzV0XpPzfDoVs4JXj+RKdFphqwjhI4XgcbT9QQcyDKSPnk5yc/gdsc5CE0AplWmAsf S86zfdnM422LRpYCNCqcUQRp5yxibOxOWVdq16cguEDU9vKQeIDcK+Z4mNMVtuLQNTVR6qoHy ak/MnATXZOE3jQ87QB4JVcpzrnvU8wuQXVNGZMAQwRWh86/JvCRo3xV+FSLhMPO5HwIiLQfZk W30pt5/zG42QXBGsjEqQz3gREGpn5Ea96lF106MfALTxDOKOMJCMD0Tv3wZyUHeHveIqAaOJI dhGJnIybtPMsFcdYY0WAzlliLBQX3H7GvF6ADWL+dHWlAY1I38E3jbfL02jUZ1gJ02ZUFh+Yy 25gIhIg8FGTBRgtWAwDUCK2NtMoWDbQR9rNk8wq3fQb6f54QcIUJePpfHjttyQBv7Bn4JeDnT d4kY+RTCMaHHGEkjbxenRDyNzQV+XxWA3QNdNGbvSNvZQUFTqjQA4f02t+oJxWSguLMlnVnYi VTWW7eDneF7hQ7ITqjWFjuTlrHp/KzJlcjB1Bnd1K9242oD0U0Tq2Ry2XLKhrJuey5a5+xxgn fr2CwfylC7vcPdBZ+9h+Eh5HIVZtdK+m1JgKA0QzXCSaek/3Lz3FfPISDjKuhmAULSFmcrzYW xBOO1bKp5JDGRr4odZcYklmvQPgdPJgEgx3U2GSwGta/SgTfeUI0SSh5g7mmVYc5/w06iHsL1 jOd0X+pRMsL28o9nm
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/ipPM5S6tMzUvDwWYptCap-G8NsY>
Subject: Re: [tsvwg] draft minutes from interim
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Mar 2020 08:12:14 -0000

Hi All,

Summary: Focussing only on raw time courses or aggregate statistics is not going to cut it, looking at both/all levels is required.

more below,


> On Mar 6, 2020, at 18:27, Bob Briscoe <in@bobbriscoe.net> wrote:
> 
> Wes,
> 
> 
> On 05/03/2020 03:04, Wesley Eddy wrote:
>> Draft minutes from the interim are available at:
>> 
>> https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/materials/minutes-interim-2020-tsvwg-01-202002200900 
>> 
> Thx for these. Look like a pretty good summary.
> 
> For those who might want to find these meeting materials in future without saving this specific URL, interim meetings are linked off the data tracker IETF proceedings page:
> https://www.ietf.org/how/meetings/interim/
> 
>> Please send any corrections or improvements.
> During Jonathan's slides, there was a conversation that seems to be completely missing where Greg asked for SCE scenarios to not just show experiments as selected individual time series that could be "cherry picked", but to usually give results over a range of realistic link rates and RTTs and a range of traffic scenarios. Also there was a request to present summary "figures of merit" e.g. queue delay, utilization, rate ratio, packet drop, ECN marking, normalized flow completion time, etc. Also to show "whiskers", e.g. 99-%ile as well as mean, across a number of experiment runs.


	[SM] I want to add, that comprehensive tests are obviously a good thing, but that by only looking at aggregate statistics and log-scales it is quite easy to miss catastrophic failures like the dualQ fairness breakdown at short RTTs. This result has been there pretty much from the beginning and yet it has not received enough attention, presumably because it was buried in big tomb of data and tests.
	Why do I keep harping on that so much? Because this failure indicates that dualQ can be gamed, any sufficiently unresponsive paced flow will be able to effectively slow everything else down to a trickle, and flows in the non-LL queue will not even be able to "fight back" with an equal amount of unresponsiveness. And IMHO that is considerably worse than the status quo, where equally responsive flows will end up sharing roughly equitable. Insisting that the failure as currently observed only happens in a "corner-case" completely ignores that (in adddition to the fact that that corner-case, short RTT flows is exactly where the CDNd internet has been moving closer and closer to).

	In short both comprehensive presentation of aggregate summary statistics for different cases as well as detailed presentations of time-resolved identified problematic cases seem appropriate.
	Regarding statistical reporting, in box whisker plots, as appropriate for expected non-normal data like delays, the mean is far less relevant than the median... also great idea to define what the whiskers are supposed to represent. The "figures of merit" idea, I believe is not unproblematic, as it tempts to only look and compare aggregate 2nd level statistics which can occlude what is actually happening at the "raw" data level. As always, one needs to also look at the raw time courses if only to avoid falling into common GIGO traps.


> 
> (The conversation started by audio, then continued on the chat). Time series are useful to dig into particular anomalous behaviours, but not really suitable for checking the operating range or to compare solutions over a range of scenarios. Higher percentile figures are essential for queue delay to determine how a real-time app would cope.

	[SM] Let's be clear, CDF type plots for delay are relevant and have merit, but the idea that you seem to be pushing here, that L4S will allow to deliver arbitrary tight real-time demands over the wider best-effort internet needs to be quashed, because it will not. Sure L4S might push down the average queueing delay over a short path down far enough to allow some tighter real-time demands than before... but it will not the end-all of queueing delay and jitter. The moment you replace the traditional bad >100ms queueing delay under load traffic shaper at the end-user/ISP transition all other variances < 100ms will start to matter...


> 
>> 
>> We are planning to get the WebEx recording posted to the IETF YouTube channel shortly, but need to do a little bit more work on that first.
> Thank you.
> And the chat log hopefully?
> 
> Now I've seen some hints at the conversation I couldn't hear during the chairs slides (audio problems), I'll start following-up on the tsvwg list.
> 
> Cheers
> 
> 
> Bob
>> 
>> 
>> 
>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/
>