Re: [tsvwg] [tcpm] L4S status tracking

[Sebastian Moeller <[e-mail]>]

Dear Bob,
 
more below in-line.

> On Dec 18, 2019, at 02:11, Bob Briscoe <[e-mail]> wrote:
> 
> Sebastian,
> 
> On 13/11/2019 12:10, Sebastian Moeller wrote:
>> (I note that the 15ms target of the 1/sqrt(p) queue does not help in this regards, why was 15 ms selected, according to codel theory 5ms should be sufficient, or is the rationale for PIE-style AQMs different and if yes why?).
> CoDel theory?

	[SM] Short hand for the "theory" sections in the codel RFC (https://tools.ietf.org/html/rfc8289#page-8) "3.2.  Target Setpoint", which is purely based on TCP behavior and independent of any AQM.

> 
> The target delay of an AQM is chosen as a tradeoff between low queuing delay and high utilization.

	[SM] Yes, with theory predicting that decent throughput/latency tradeoffs require target to be in the range of 5-10% of the RTT. In practice it seems to be sufficient for interval to be in the right order of magnitude though.


> As below...
> 
> For an AQM on an access link, there will not be a high degree of statistical multiplexing, so it has to work well for a single flow.

	[SM] It should work reasonable for a single flow, but even speedtests have more or less all switched to employing multiple streams so that maximizing single stream throughput at all costs seems like an idea past its due date.

> For full utilization, a Reno flow needs 1 BDP of buffer (i.e. if measuring buffer in time, the same amount of buffer as the the base RTT). So, if an operator configures CoDel (whether FQ or not) with a target of 5ms, a Reno flow will underutilize the link for all paths with a base RTT greater than 5ms.

	[SM] That is not how target works, both PIE ad CoDel have two relevant concepts, the "permissible" standing queue delay or "target" and the expected reference RTT, "interval" in CoDel, reference latency or "ref_del" in PIE. It is that later parameter that allows for transient buffering of packets until a flow meaningfully responds to the AQM's signaling. For CoDel, in short, if the delay of the standing queue stays above target for at least interval, CoDel will consider sending a slow-down signal. PIE, I believe acts similarly.


> The utilization will be:
>     75% + target/baseRTT * 25%
> For example, if target = 5ms and baseRTT=25ms, utilization will be
>     75% + 5/25 * 25%
>     = 80%
> So if you run speedtest.net in single flow mode to a server showing a ping time =25ms, you won't see more than about 80% of your link capacity.

	Sure, and for Reno that is actually 5% better than can be expected from naively looking at the actual saw tooth pattern, which would predict only 75% of link capacity used.

> (It's not quite as bad as that 'cos speedtest.net uses Cubic, and Cubic under-utilizes a bit less badly than Reno with a shallow queue threshold - the formula is much more complex tho.)

	[SM] Well, almost everybody uses the default multi-stream test mode anyways...

> 
> Therefore, assuming most users care about latency, but also don't want to lose much capacity, it makes sense to configure target to a typical base RTT.

	[SM] No, that is not how target and interval are recommenced to be set. Interval should be in the order of magnitude of expected RTTs, while target should be in the range of 5-10% of interval for TCP (and hence also for tcp-friendly) flows.

> Most users nowadays talk predominantly with CDNs, and in a large city environment that probably means a typical RTT is maybe 10-20ms (finger in the air).

	[SM] Yes, that is exactly why I consider the dualQ target of 15ms to be massively misguided, at that RTT target theoretically should be 1 to 2 milliseconds. The default 15 ms seems taylored for RTTs of 300ms.

> 
> The designers of CoDel recommended 5ms, I assume on the basis that they believed removing another 5-10ms of delay was worth trading off against 10-20% more under-utilization.

	[SM] Please read rfc8289 again, the rationale is laid out there quite compellingly, no need for speculation. 

> 
> The designers of PIE recommended 15ms, I assume on the basis that they believed users would not be happy if their speedtest results sufered, even if their latency improved.

	[SM] And again you assume, let's look at the sequence of references:

https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10:

"*  for the PI2 algorithm (Appendix A) the queuing delay target is
         set to the typical RTT;"

"For link rates from 4 - 200 Mb/s, a
   target RTT of 15ms and a maximum RTT of 100ms, it has been verified
   through extensive testing that Tupdate=16ms (as recommended in
   [RFC8033]) is sufficient."


Appendix A
"   7:    % PI2 AQM parameters
   8:    RTT_max = 100 ms                      % Worst case RTT expected
   9:    RTT_typ = 15 ms                                   % Typical RTT
   11:   % PI2 constants derived from above PI2 parameters
   10:   p_Cmax = min(1/k^2, 1)             % Max Classic drop/mark prob
   12:   target = RTT_typ            % PI AQM Classic queue delay target"


No justification for RTT_typ, but a reference to https://www.rfc-editor.org/rfc/rfc8033.html

"This document presents a lightweight active queue management design
   called "PIE" (Proportional Integral controller Enhanced) that can
   effectively control the average queuing latency to a target value."

"      -  QDELAY_REF.  AQM Latency Target (default: 15 milliseconds)"

Note how dual queue draft, seemingly misinterpreted QDELAY_REF categorically as being one of the actually encountered RTTs instead of what it truly is a limit for acceptable/average standing queue delay. 


There is no further justification for the setting of QDELAY_REF = 15 ms except a reference to https://www.researchgate.net/publication/261134127_PIE_A_lightweight_control_scheme_to_address_the_bufferbloat_problem?origin=mail

Which says exactly nothing on the selection of ref_del, but seems to set it either at 20 or 5 milliseconds without expanding on how to select it. It is clear that that here ref_del is not directly related to the RTT:

"There is a subtle difference in the definition of terms: in CoDel, the parameter, target, represents the target latency, while del ref refers the reference equilibrium latency in PIE. For easy comparison, we would use ”Reference Delay” to refer the target latency, target, in CoDel and the reference equilibrium latency, del ref, in PIE."


Comparing PIE and CoDel with 20 TCP-flows @100ms RTT 10Mbps link we get:

target	throughput PIE [Mbps]	throughput CoDel [Mbps]
20		9.87					9.94
5		9.66					9.84


Note how the throughput sacrifice for going from 20 ms to 5ms is only 2% for PIE and 1% for CoDel. Just re-reading this paper again, it seems obvious that PIE has no inherent issues why target (for RTT 100ms flows) needs to be larger than 5ms.


Looking at these numbers, I would very much like to invite the dual queue coupled AQM implementers to test again with target set to 5ms and see whether that ameliorates the gross imbalance at short RTTs between both of L4S queues.
Alternatively, I would be happy for a link to a paper/data demonstrating that target=5 has unacceptable side-effects. This is especially relevant since the rate imbalance introduced between the two queues will make the standard-compliant queue suffer far worse than a drop of 2% points at short RTTs.


> 
> There is also a difference between the meaning of the target in CoDel and PIE. From memory of [Kuhn16], I think it was that CoDel's target is broadly the min, whereas PIE's target is the average.

	Yes they are not identical,  but CoDel's target really describes the amount of queueing delay one is willing to tolerate at saturation, and low and behold, ref_del for PIE pretty much also describes the average standing queue delay under saturating conditions, so they are broadly similar. Similar enough to merit testing whether giving the standards compliant queue in L4S a saner target can paper over dual queue coupled's gross failures to properly isolate the two traffic types from each other. Don't get me wrong, I would very much prefer that you go and fix dualQ to actually work robustly and reliably independently of whether flows in any of the two queues play nicely or misbehave, but at the least you should look how your choice of target affects systemic queue imbalance at short RTTs. But by all means keep arguing why target 5ms will not work instead of running tests and figure out whether that is actually true or not. Best way to shut me up, is actual data...

Best Regards
	Sebastian

P.S.: The tone might still be adversarial, but I hereby contribute literature research to support my proposal to change dialQ's target to 5ms in an attempt to also improve the latency under load experienced by non-L4S traffic, as we all can agree that low-latency is a worty goal to strive for for all flows.



> 
> 
> 
> [Kuhn16] Nicolas Kuhn, David Ros, Amadou Baba Bagayoko, Chamil Premantha Kulatunga, Gorry Fairhurst, Naeem Khademi "Operating ranges, tunability and performance of CoDel and PIE"
> 
> 
> 
> Bob
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>