Re: [tsvwg] [tcpm] L4S status tracking

Sebastian Moeller <> Wed, 18 December 2019 10:25 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 0D7D41200FD for <>; Wed, 18 Dec 2019 02:25:26 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.649
X-Spam-Status: No, score=-1.649 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id sxkSoqZip7HA for <>; Wed, 18 Dec 2019 02:25:24 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id BC15A1200F7 for <>; Wed, 18 Dec 2019 02:25:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1576664721; bh=wPu9mggmm1QsByafKVYpSKjMeutEQfv0f4VJwu4pcEQ=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=b+PHMES5JrDU+1/yFG+eC4cpldxs3wcGxi0YOh5SM9p3TLsnW6utosBAxjrngSBPG s2Df2ElZzwEITY5Wdmgw+fODaujWc1GPteuTQlwT6LWx4Chgjyut/2jyvQnZTlvyz5 D0qA7+y34IplS+KWYeL4W7O5W8fbStNCk5J4dRfg=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (mrgmx004 []) with ESMTPSA (Nemesis) id 1MacOW-1i6WyE2Log-00cAto; Wed, 18 Dec 2019 11:18:52 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Wed, 18 Dec 2019 11:18:50 +0100
Cc: "Scharf, Michael" <>, "Rodney W. Grimes" <>, tsvwg IETF list <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <> <> <> <> <> <> <> <> <> <>
To: Bob Briscoe <>
X-Mailer: Apple Mail (2.3445.104.11)
X-Provags-ID: V03:K1:BRcsf9tmYKdy7Io0Zu50url5VBN6KNkhYqnTu12skNi90aYtFE+ WKlFGIYwtWyytm6w8+pca/3LimN8K+HLi5QooCLHPFgg6rL2nmePkmIhxz6YzsSeHt80age poSz9uV1Qqm4/cxwdcObdMz3PBN54Sr7QuqCkiczPxGFjPCcJaPJXJF9O/SXEJ2TUgO8kK1 gKlnCfJiC047N/SedyFXw==
X-UI-Out-Filterresults: notjunk:1;V03:K0:Kuy+GAEVJr4=:MhFMW25+5EelLyUG6Oc63C GfZBLQ5R6rr0p/cd1Z5qtUpygghizbu12QJE04jSAms6LB/OHJFnBrRD85jFb0Vhw/re/vKfX Kf7uXjPfJOfCri1WSmA91tBzPSZb4g2kcy2jJOUhGdffIXUJ67MFioVl2LvEKzesXaZxYSxlY +bnt0jWcuMZ1B+OZtkvQi497q6KZ0qNKO8AL0SpHGN2KeZ22hdHWtUlFkRXhKABM949mClP9O vv+WMckN2osCwygaEQlkD+prnAjzcqbW3FOeAlP5W6L+f+pDUqQu3OYRhH4MHA8CqeHCHRIDM sPmQvDS3hIcRD7yTt+JCCfrvBxBXF9tDw+HfFAMBBBifkiVD05DUX8gxGvqXcd8xRDtZ9WN7T 652ITLuQbMYHQy8g1RDBrma/mqTvAPoZEh0cXWKmAnaUqnrVMcrLC7ltTOi25CTav1zKWohxy ZlLUz8e1aQCdQSuz0UYPNyvac1CnAa9mZSdi+SSWvhsNPuQIojXcpI+063Z0t9wM5QXMw7IuN DZvvMgiHWudOQqQcJdJvRkgawzcRBhiRYoW1kgeCgZQhF15muBfi+6mBMKG7xtPbVy5ITDEic ec7d1v7zgVtQpwoEumQEnOk97Hy7/O1boXpSNiuFi16llT8y/q6HbOUJ1rzSzAbFB0t87BkpN OhxJLsHbZOOZEM62Z2ziOgI7Z8CJLj3RaPf0mPPAf1bFZuG/aomdy4a9YbCtg+3ze8mrzetS2 pSYLL73sxHCR9oNtOQmi3ZVJHCLs2tZOlQfKrYvyZVm0pMluQOz9IXznvweOpDRfkgNmxpmB7 +2QRTzfQaGkloNfaqLBIoWlOL5h7r5HMbTG7xYDe5qB0VErkcaG5lfdjAIB42aPERb4HEWqUe ETiRqckECRPkNf5I1KlSqU7HitzXqigPcn8dhMWlDuNTiiXhvzvL/G0I7BiHb42qxmgqcWyef 90XEVhkBsi2MEvIin4gf4KipoeyDr5pvyCAC6d+jHM4XuC/USfSxB+wlmcRfwEqmudjcdnjK8 +1MDTiWZAMtHeVwIOYehd6ccKoqR6wAk9WPKNWEeMI9CCVQ07g0qvClmevbnuNA3Ec0144bpG spiK1cJ/ztQMvr6fs0Qa64IR1PvwOp7RkLvvewlQJTXuv0dBbjySe7l3lKVIkDXembXIpTG/3 wtlA+MUFdUFVbSoTWB4Exf+ejyvMKiPTkt8F33yyMvRtKsV0W0TQJim79ON1AeYJjFPfFIG3d jjGeOWn0QOc/iLoZYsy4/1ckjW2ifmFuVLRXrXw==
Archived-At: <>
Subject: Re: [tsvwg] [tcpm] L4S status tracking
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 18 Dec 2019 10:25:26 -0000

Dear Bob,
more below in-line.

> On Dec 18, 2019, at 02:11, Bob Briscoe <> wrote:
> Sebastian,
> On 13/11/2019 12:10, Sebastian Moeller wrote:
>> (I note that the 15ms target of the 1/sqrt(p) queue does not help in this regards, why was 15 ms selected, according to codel theory 5ms should be sufficient, or is the rationale for PIE-style AQMs different and if yes why?).
> CoDel theory?

	[SM] Short hand for the "theory" sections in the codel RFC ( "3.2.  Target Setpoint", which is purely based on TCP behavior and independent of any AQM.

> The target delay of an AQM is chosen as a tradeoff between low queuing delay and high utilization.

	[SM] Yes, with theory predicting that decent throughput/latency tradeoffs require target to be in the range of 5-10% of the RTT. In practice it seems to be sufficient for interval to be in the right order of magnitude though.

> As below...
> For an AQM on an access link, there will not be a high degree of statistical multiplexing, so it has to work well for a single flow.

	[SM] It should work reasonable for a single flow, but even speedtests have more or less all switched to employing multiple streams so that maximizing single stream throughput at all costs seems like an idea past its due date.

> For full utilization, a Reno flow needs 1 BDP of buffer (i.e. if measuring buffer in time, the same amount of buffer as the the base RTT). So, if an operator configures CoDel (whether FQ or not) with a target of 5ms, a Reno flow will underutilize the link for all paths with a base RTT greater than 5ms.

	[SM] That is not how target works, both PIE ad CoDel have two relevant concepts, the "permissible" standing queue delay or "target" and the expected reference RTT, "interval" in CoDel, reference latency or "ref_del" in PIE. It is that later parameter that allows for transient buffering of packets until a flow meaningfully responds to the AQM's signaling. For CoDel, in short, if the delay of the standing queue stays above target for at least interval, CoDel will consider sending a slow-down signal. PIE, I believe acts similarly.

> The utilization will be:
>     75% + target/baseRTT * 25%
> For example, if target = 5ms and baseRTT=25ms, utilization will be
>     75% + 5/25 * 25%
>     = 80%
> So if you run in single flow mode to a server showing a ping time =25ms, you won't see more than about 80% of your link capacity.

	Sure, and for Reno that is actually 5% better than can be expected from naively looking at the actual saw tooth pattern, which would predict only 75% of link capacity used.

> (It's not quite as bad as that 'cos uses Cubic, and Cubic under-utilizes a bit less badly than Reno with a shallow queue threshold - the formula is much more complex tho.)

	[SM] Well, almost everybody uses the default multi-stream test mode anyways...

> Therefore, assuming most users care about latency, but also don't want to lose much capacity, it makes sense to configure target to a typical base RTT.

	[SM] No, that is not how target and interval are recommenced to be set. Interval should be in the order of magnitude of expected RTTs, while target should be in the range of 5-10% of interval for TCP (and hence also for tcp-friendly) flows.

> Most users nowadays talk predominantly with CDNs, and in a large city environment that probably means a typical RTT is maybe 10-20ms (finger in the air).

	[SM] Yes, that is exactly why I consider the dualQ target of 15ms to be massively misguided, at that RTT target theoretically should be 1 to 2 milliseconds. The default 15 ms seems taylored for RTTs of 300ms.

> The designers of CoDel recommended 5ms, I assume on the basis that they believed removing another 5-10ms of delay was worth trading off against 10-20% more under-utilization.

	[SM] Please read rfc8289 again, the rationale is laid out there quite compellingly, no need for speculation. 

> The designers of PIE recommended 15ms, I assume on the basis that they believed users would not be happy if their speedtest results sufered, even if their latency improved.

	[SM] And again you assume, let's look at the sequence of references:

"*  for the PI2 algorithm (Appendix A) the queuing delay target is
         set to the typical RTT;"

"For link rates from 4 - 200 Mb/s, a
   target RTT of 15ms and a maximum RTT of 100ms, it has been verified
   through extensive testing that Tupdate=16ms (as recommended in
   [RFC8033]) is sufficient."

Appendix A
"   7:    % PI2 AQM parameters
   8:    RTT_max = 100 ms                      % Worst case RTT expected
   9:    RTT_typ = 15 ms                                   % Typical RTT
   11:   % PI2 constants derived from above PI2 parameters
   10:   p_Cmax = min(1/k^2, 1)             % Max Classic drop/mark prob
   12:   target = RTT_typ            % PI AQM Classic queue delay target"

No justification for RTT_typ, but a reference to

"This document presents a lightweight active queue management design
   called "PIE" (Proportional Integral controller Enhanced) that can
   effectively control the average queuing latency to a target value."

"      -  QDELAY_REF.  AQM Latency Target (default: 15 milliseconds)"

Note how dual queue draft, seemingly misinterpreted QDELAY_REF categorically as being one of the actually encountered RTTs instead of what it truly is a limit for acceptable/average standing queue delay. 

There is no further justification for the setting of QDELAY_REF = 15 ms except a reference to

Which says exactly nothing on the selection of ref_del, but seems to set it either at 20 or 5 milliseconds without expanding on how to select it. It is clear that that here ref_del is not directly related to the RTT:

"There is a subtle difference in the definition of terms: in CoDel, the parameter, target, represents the target latency, while del ref refers the reference equilibrium latency in PIE. For easy comparison, we would use ”Reference Delay” to refer the target latency, target, in CoDel and the reference equilibrium latency, del ref, in PIE."

Comparing PIE and CoDel with 20 TCP-flows @100ms RTT 10Mbps link we get:

target	throughput PIE [Mbps]	throughput CoDel [Mbps]
20		9.87					9.94
5		9.66					9.84

Note how the throughput sacrifice for going from 20 ms to 5ms is only 2% for PIE and 1% for CoDel. Just re-reading this paper again, it seems obvious that PIE has no inherent issues why target (for RTT 100ms flows) needs to be larger than 5ms.

Looking at these numbers, I would very much like to invite the dual queue coupled AQM implementers to test again with target set to 5ms and see whether that ameliorates the gross imbalance at short RTTs between both of L4S queues.
Alternatively, I would be happy for a link to a paper/data demonstrating that target=5 has unacceptable side-effects. This is especially relevant since the rate imbalance introduced between the two queues will make the standard-compliant queue suffer far worse than a drop of 2% points at short RTTs.

> There is also a difference between the meaning of the target in CoDel and PIE. From memory of [Kuhn16], I think it was that CoDel's target is broadly the min, whereas PIE's target is the average.

	Yes they are not identical,  but CoDel's target really describes the amount of queueing delay one is willing to tolerate at saturation, and low and behold, ref_del for PIE pretty much also describes the average standing queue delay under saturating conditions, so they are broadly similar. Similar enough to merit testing whether giving the standards compliant queue in L4S a saner target can paper over dual queue coupled's gross failures to properly isolate the two traffic types from each other. Don't get me wrong, I would very much prefer that you go and fix dualQ to actually work robustly and reliably independently of whether flows in any of the two queues play nicely or misbehave, but at the least you should look how your choice of target affects systemic queue imbalance at short RTTs. But by all means keep arguing why target 5ms will not work instead of running tests and figure out whether that is actually true or not. Best way to shut me up, is actual data...

Best Regards

P.S.: The tone might still be adversarial, but I hereby contribute literature research to support my proposal to change dialQ's target to 5ms in an attempt to also improve the latency under load experienced by non-L4S traffic, as we all can agree that low-latency is a worty goal to strive for for all flows.

> [Kuhn16] Nicolas Kuhn, David Ros, Amadou Baba Bagayoko, Chamil Premantha Kulatunga, Gorry Fairhurst, Naeem Khademi "Operating ranges, tunability and performance of CoDel and PIE"
> Bob
> -- 
> ________________________________________________________________
> Bob Briscoe