Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

["De Schepper, Koen (Nokia - BE/Antwerp)" <[e-mail]>]

Hi Sebastian,

>> Actually, I'm also not happy with this strong RTT dependency. You probably will see similar unfairness cases if you use under these conditions Reno on a CoDel AQM with a target of 5ms.
>
>	[SM] Have a look at the fq_codel:
>

I really meant "CoDel", not "FQ_CoDel".




> But I doubt the Linux Cubic developers would be happy to extend Cubic to avoid CoDel flaws??
>
>	[SM] Too late, codel is out in the wild, no amount of whining is going to change that, if you design a protocol for the existing internet you need to accept that fact and adapt to it.

So because CoDel it is deployed and you get these bad results today with existing Cubic deployments already, it is not a safety issue, but whining is appropriate???? It clearly shows that this "bar" is kept very high for some and is laying at the ground for others 😉.



Koen.



-----Original Message-----
From: Sebastian Moeller <[e-mail]> 
Sent: Monday, November 16, 2020 10:37 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]>
Cc: Pete Heist <[e-mail]>; Black, David <[e-mail]>; tsvwg IETF list <[e-mail]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

Hi Koen,


> On Nov 16, 2020, at 21:43, De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]> wrote:
> 
> Hi Pete,
> 
> Thanks for the tests. They do confirm our test results from many years ago now, and are the reason why related Prague requirements are in the L4S-ID draft. I guess they are a good introduction for people recently joining the discussions and a reminder for the rest that it is important to take those conditions into account when designing and configuring both the end-host and the network mechanisms. 

	[SM] Please supply a link to your results from many years ago showing how the current state of TCP Prague behaves? 


> 
> May I also remind that the Linux Prague code is research driven, and focused only on de-risking the bigger safety challenges.


	[SM] Papering over obvious flaws in the core L4S design at a place that can be easily abandoned later would be a much less friendly interpretation.


> Most of the improvements (except low latency) were not prioritized up 
> to now,

	[SM] Mmmh, did you not just argue you focussed on safety challenges first?

> as we are sure some are low hanging fruit and others with the right incentives will want to spend time on the harder ones.

	[SM] The fact that TCP Prague has terrible performance over FIFO-bottleneck long haul paths will make it very unlikely that someone will be incentivized to continue TCP Prague development after you abandon it.

> Most of the commercial parties with an expected interest and benefit 
> of an L4S CC have been reasonable quite,

	[SM] Specifically none responded for my repeated begging for presenting data to demonstrate that my fears are wrong. My conclusion from that lack of response is that either nobody cares for L4S or worse nobody bothered to independently confirm its suitability and safety.


> I hope because they are confident that the network drafts and the mandatory Prague requirements and proposed improvements are feasible. I guess some are also less keen on sharing their efforts with competition and others might be waiting for real deployments before getting really into action. If there are concerns from their side, of course we would like to hear those too. The only concern I heard up to now is that all this takes too long before they can start the experiment. They want asap real world deployment...

	[SM] There is zero rationale why a controlled experiment could not have already been performed over the existing internet. An experimental RFC is not a requirement to conduct an experiment. What you call an experiment, really is an exercise in deployment.


> 
>>> 1) Starting with an easier one, can the reported intolerance to 
>>> bursts
>>> (https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by starting marking in the L queue later? Doing so may also improve RTT fairness for Prague flows.
> 
> Indeed, the solution is to make sure that your L4S threshold configuration is in line with your network capabilities. If you know that certain network elements aggregate and burst packets in 4ms chunks, then it makes no sense at all to mark packets in that node at 1ms, nor in subsequent network nodes, with similar bottleneck capacities. We tried to address this by explaining this in section 6.3 of the L4S Architecture draft: https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-08#page-20. Is this sufficient to cover this issue?

	[SM] What happened to the disire for AQM's to be auto-tuning to avoid the biggest stumbling block for their deployment? Think https://tools.ietf.org/html/rfc7567#page-18 "4.3.  AQM Algorithm Deployment SHOULD NOT Require Operational Tuning".



> 
>>> 2) For the reported network bias, can this be fixed in the DualPI2 and Prague implementations? One chart which we didn't reference in our writeup illustrates the consistent bias for Prague vs CUBIC.
> 
> Since an important design goal of L4S is that it also supports network nodes that don't identify individual flows.

	[SM] Well, it seems more and more obvious that that design decision puts L4S at odds with achieving its goals....


> To compensate individual flow's RTT you need to know them first. To solve this anyways, it would need a per packet indication of the RTT (ref past proposals like RCP and XCP). If we would have space to put this info in the header, we wouldn't be discussing about this single available ECN bit, right? Therefor the End-System is the right place to handle this, and why the RTT-independent requirement is important.

	[SM] Please keep your story straight, you just proposed to considerably water down the RTT de-bias requirement such, that it is not really applicable to arbitrary paths anymore, only those that the AQM deployer cares for, like a fast-lane for short RTTs.



> The current implementation has a non-Prague-compliant default (not RTT-independent), which we need to fix clearly. It would be good to agree on a reasonable compromise of a default f(RTT) correction function. We also implemented a gradual correction over time, so a new (short) flow would not be delayed, and will be responsive, only if it lasts for a while it will gradually converge to the RTT-independent rate. This could be in the range of the convergence time of the typical-RTT flows.

	[SM] As I have shown with Pete's data in my response to Bob your problem is not the to be expected RTT dpendence of the thoughput of competing TCP flows, but the combination of DualPI2 and TCP Prague actually makes the situation much worse, see (repeated from my other mail):

Let's look at some data, shall we?
This is a plot showing the rates achieved by two CUBIC flows (80ms pRTT and 20 ms pRTT) in a bottleneck with a 200 packet FIFO at an egress rate of 50 Mbps http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-pfifo_200_-10Mbit-80ms-20ms_tcp_delivery_with_rtt.svg
Both flows actually get more or less the same thoughput.
	Now adding an AQM we expect things to get worse, especially single queue AQMs are known to do that, so here are "same" two CUBIC flows under the same conditions only the bottleneck now operates DualPI2 but since the flows are not ECT(1) they are handled by the PIE instance responsible for the shallow queue:
http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-dualpi2-50Mbit-80ms-20ms_tcp_delivery_with_rtt.svg
Here eye-balling it tells us that the 80ms pRTT flow only gets around half the thoughput of the 20ms one, not beautiful, but certainly acceptable by the concurrent decrease of the flows' queueing delay (roughly a reduction of 230ms per flow).
	But now, if we look at the same test with two TCP Prague flows running through the shallow-queue's modified? PIE instance we see a throughput imbalance of around 45:5 Mbps or 9:1.
	So from rough parity we went down to 9:1, that IMHO is a clear design issue in L4S because who in their right mind is going to use TCP Prague if that basically means such steep sacrifices in throughput through longer paths? Really, it is not that 80ms is an extreme RTT that is irrelevant, think trans Atlantic links.



	Trying to frame this as somehow reducing RTT bias seems simply counterfactual, L4S is making the situation worse (while using wording that makes the naive reader think it helps solving the issue).


> 
> Feel free to experiment with the available options.
> 
>>> * To not consider a 16:1 throughput imbalance between L4S and non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
>>> Although we're not sure of the worst case, what we're seeing now is outside of my comfort zone, personally.
> 
> Actually, I'm also not happy with this strong RTT dependency. You probably will see similar unfairness cases if you use under these conditions Reno on a CoDel AQM with a target of 5ms.

	[SM] Have a look at the fq_codel:

Here is how the dual CUBIC test looks in fq_codel:
http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-fq_codel-50Mbit-80ms-20ms_tcp_delivery_with_rtt.svg
Way less than the factor two throughput difference as seen in your PIE example. Judging from Figure 6 in Høiland-Jørgensen, Toke, Per Hurtig, and Anna Brunstrom. ‘The Good, the Bad and the WiFi: Modern AQMs in a Residential Setting’. Computer Networks 89 (4 October 2015): 90–106. https://doi.org/10.1016/j.comnet.2015.07.014. single-queue CoDel will behave quite close to songle-queue PIE.


Your intuition seems to be wrong.



> Cubic compensates performance for the longer RTT flows, Prague (with the right settings) compensates mandatory for the smaller RTT. Clearly a production congestion control is likely to do both, or fall back to Classic Cubic behavior whenever the RTT is too long for interactive services.

	[SM] Well, TCP Prague is your vehicle to demonstrate to the world that your claims are achievable, so go and achieve them. Until that point all siuch claims are really just predictions and promises

> 
> So the exact out-of-our-confort-zone exists today already, but there are solutions available, just to be integrated.
> 
>>> 3) The issues that arise due to the redefinition of CE may be the 
>>> hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes)
> 
> This is fortunately showing the flaws of CoDel and on top of that the lack of the CoDel implementation to respond with drop when ECN flows get into overload. Even more problematic for Cubic on CoDel (deployed today), your results show it is also bad (more than 2 seconds latency for longer than 10 seconds when throughput reduces from 50Mbps to 1Mbps on an 80ms base RTT path). When compared under the same conditions with Cubic on PI2 you can definitely see how it can be improved using an appropriate AQM (a very short peak of I guess less than 100ms extra latency). Hopefully these CoDel instances can be fixed and can support the ECN threshold for L4S instead (and maybe use PI2 instead of CoDel for Classic). On the other hand these extremes would be also easy cases for both Classic and L4S CCs to detect and fall back on a delay based CC when latencies are that bad (falling back to Classic ECN or Classic drop would even not help to avoid the CoDel problem). But I doubt the Linux Cubic developers would be happy to extend Cubic to avoid CoDel flaws??

	[SM] Too late, codel is out in the wild, no amount of whining is going to change that, if you design a protocol for the existing internet you need to accept that fact and adapt to it.


> 
>>> Although I won't be able to support a WGLC until I see tested code that addresses the issues, I do want to support the WG along its present path to a conclusion...
> 
> Try switching on the RTT-independent settings of Prague, and I assume it is a matter of just coding a fallback to Cubic if your detected RTT is too big. So as far as I'm concerned, we do have running code! 

	[SM] The idea behind this requirement is to have code that can be executed (runs) but also solves the problem at hand. TCP Prage as from the default repository might do the former, but as Pete showed fails at the latter. 


> 
> I'll try to get the defaults right. I assume f(RTT)=max(25ms, RTT) and a gradual EWMA convergence reaching the 25ms behavior after about 10 seconds would be good defaults? If people have suggestions or preferences for other values, then let me know. If people have time and want to implement the Cubic fallback (or even better longer RTT-independence), then let me know too.

	[SM] Think auto-tuning, what you do is window dressing, by modifying your toggles to fit a specific individal benchmark you look bad at, but unless TCP Prague does that adaptively roughly correct, this is no viable solution, that can be used as argument that the other deeply flawed L4S components are safe to deploy.


Regards
	Sebastian


> 
> Regards,
> Koen.
> 
> 
> -----Original Message-----
> From: tsvwg <[e-mail]> On Behalf Of Pete Heist
> Sent: Sunday, November 15, 2020 12:42 PM
> To: Black, David <[e-mail]>; Sebastian Moeller 
> <[e-mail]>
> Cc: tsvwg IETF list <[e-mail]>
> Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow 
> latency
> 
> Hi guys, and thanks for the replies.
> 
> Firstly, there was a new test result added for tunnels yesterday, but I'll put that in a separate thread so it isn't buried.
> 
> Beyond that, I think it would be productive to try to figure out what might be done to solve the reported issues, limiting that exploration for now to the confines of the L4S design. We'd like to support forward progress while facing the issues head on if possible. I'll take a shot here as tersely as I can:
> 
> 1) Starting with an easier one, can the reported intolerance to bursts
> (https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by starting marking in the L queue later? Doing so may also improve RTT fairness for Prague flows.
> 
> 2) For the reported network bias, can this be fixed in the DualPI2 and Prague implementations? One chart which we didn't reference in our writeup illustrates the consistent bias for Prague vs CUBIC.
> Essentially, Prague wins until there is an RTT imbalance of 20ms/80ms:
> 
> http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/s1-charts/l4
> s_network_bias.svg
> 
> 3) The issues that arise due to the redefinition of CE may be the hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes) 
> 
> Afaik, the proposed solutions in the L4S architecture are:
> 
> * RFC3168 bottleneck detection in the endpoints. However, we went through a round of testing earlier this year that showed accurate bottleneck detection is likely to be difficult with jitter and cross- flow traffic. It's disabled at present in the reference implementation.
> 
> * L4S operational guidance for network operators. So far I haven't had time to review this, but I may have some feedback at least in the area of testing. I suspect the effectiveness of guidance will be influenced by human factors.
> 
> * To not consider a 16:1 throughput imbalance between L4S and non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
> Although we're not sure of the worst case, what we're seeing now is outside of my comfort zone, personally.
> 
> 
> For me, the class of problems in #3 are my area of greatest concern, as there are many more RFC3168 bottlenecks deployed today than just a few years ago (https://github.com/heistp/l4s-tests/#deployments-of-fq_codel) #2 is also important to me, as I trust we're not trying to introduce a so- called "fast lane" for certain traffic.
> 
> Although I won't be able to support a WGLC until I see tested code that addresses the issues, I do want to support the WG along its present path to a conclusion...
> 
> Pete
> 
> On Sun, 2020-11-15 at 01:26 +0000, Black, David wrote:
>> Hi Sebastian,
>> 
>> Some comments as an individual, not a WG chair.
>> 
>> First, I think Pete has pretty clearly established that TCP Prague is 
>> research, and hence (IMHO) TCP Prague ought to be headed for ICCRG as 
>> its primary forum rather than TSVWG.
>> 
>> To date, the 'Prague L4S Requirements' (Appendix A of draft-ietf-
>> tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague. That 
>> association ought to be teased apart so that the resulting L4S 
>> scalable congestion control requirements provide a reasonable design 
>> space that can include a number of other congestion control designs - 
>> in addition to what's been discussed, e.g., SCReAM, it would be 
>> useful to better understand what it would take for implementations of 
>> protocols such as DCTCP and BBR (e.g., for QUIC) to meet those 
>> requirements.
>> 
>> My overall take on the requirements is that in 20/20 hindsight, some 
>> of them were overly optimistic, and hence need to be backed off/toned 
>> down/broadened to encompass what is reasonable in "running code" well 
>> beyond TCP Prague. That sort of collision between interesting ideas 
>> and network realities is not an unheard-of scenario in IETF, so I 
>> hesitate to view the need for changes to these requirements as 
>> evidence that the original ideas were inherently defective, as I've 
>> seen far more dramatic changes, e.g., some number of years ago, the 
>> first design of iSCSI login was elegant ... and resulted in 
>> implementations that did not interoperate, resulting in a complete 
>> redesign.
>> 
>> Thanks, --David
>> 
>>> -----Original Message-----
>>> From: tsvwg <[e-mail]> On Behalf Of Sebastian Moeller
>>> Sent: Thursday, November 12, 2020 6:11 PM
>>> To: Pete Heist
>>> Cc: tsvwg IETF list
>>> Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow 
>>> latency
>>> 
>>> 
>>> [EXTERNAL EMAIL]
>>> 
>>> Hi Pete,
>>> 
>>> great data. IMHO Especially the fact that short-RTT Prague will 
>>> severely outcompete long-RTT Prague way more than the traditional 
>>> dumb FIFO will, strongly supports  the following hypotheses I have 
>>> been posting
>>> before:
>>> 
>>> a) Dualpi2/TCP Prague are no way ready for deployment, but are 
>>> basically still in the toy stage
>>> 
>>> b) all that L4S will effectively do, be it by intent or simply by 
>>> mis-design, is built a "fast lane" for short RTT low hop count 
>>> traffic. Given all the hype
>>> (still!) in the L4S
>>> internet drafts about how this is the future of the internet...
>>> Also a fast lane that
>>> requires active cooperation from the leaf ISP (if they keep their 
>>> bottlenecks at FIFO, I see no compelling reason for TCP Prague at 
>>> all), which will require SLAs between CDNs and ISPs.
>>> 
>>> c) too little, too late. Really only modest gains in RTT (compared 
>>> to best of class AQMs like fq_codel or cake) and noticeable 
>>> regressions in sharing behavior both between different CCs and flows 
>>> of different RTTs (and that compared to dumb queue "management" with 
>>> a simple FIFO).
>>> 
>>> 
>>> 
>>> Also quite sobering, that it AGAIN was not team L4S bringing real 
>>> data to the table to support their claims and promises. Then again 
>>> looking at the RTT fairness for Prague versus Prague under DualQ, I 
>>> can understand why team L4S stayed mumm, and rather argued for 
>>> allowing unfettered self-mutilation of L4S compliant transport 
>>> protocols in regards to RTT fairness:
>>> 
>>> "So there is no need  to mandate that an L4S implementer does no 
>>> harm to themselves, which window-based congestion controls tend to 
>>> do at higher RTT.
>>> Of course, this doesn't preclude implementers reducing or 
>>> eliminating RTT bias for larger than typical RTTs, but it removes 
>>> any requirement to do so."
>>> 
>>> After years of advertising on increased RTT independence (in spite 
>>> of the data showing that the proposed combination of DualQ and TCP 
>>> Prague actually increases RTT bias), team L4S in the last minute no 
>>> less, decides to do a 180 and just change the requirements to allow 
>>> for the rather unhealthy behavior demonstrated for L4S...
>>> With the current enhanced RTT bias, who in their right mind is going 
>>> to use TCP Prague; which currently is the only transport, that at 
>>> least attempted to tackle all the "requirements" L4S poses for 
>>> transports that want to use the
>>> ECT(1) express
>>> way? Then again, since none of the network elements are designed and 
>>> required to actually check and enforce these requirements, calling 
>>> these "requirements" is a bit of stretch anyway, maybe they should 
>>> be changed from MUST requirements to COULD musings....
>>> 
>>> 
>>> Best Regards
>>>         Sebastian
>>> 
>>> 
>>>> On Nov 12, 2020, at 21:31, Pete Heist <[e-mail]> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> We have posted some new tests of L4S here:
>>>> 
>>>> https://github.com/heistp/l4s-tests
>>>> 
>>>> These tests cover two-flow fairness in several qdiscs with 
>>>> different path RTTs per-flow, and transient behavior in fq_codel 
>>>> queues.
>>>> 
>>>> The results raise some concerns about a bias in favor of L4S flows 
>>>> in
>>>> DualPI2 queues, throughput imbalances between flows with different 
>>>> path RTTs, and intra-flow latency spikes upon rate reductions in 
>>>> fq_codel.
>>>> The repo above contains a walk-through of the key findings, and 
>>>> links to more results in the Appendix.
>>>> 
>>>> Regards,
>>>> Pete
>>>> 
>>>> 
>> 
> 
>