Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

["De Schepper, Koen (Nokia - BE/Antwerp)" <[e-mail]>]

Hi Pete,

Thanks for the tests. They do confirm our test results from many years ago now, and are the reason why related Prague requirements are in the L4S-ID draft. I guess they are a good introduction for people recently joining the discussions and a reminder for the rest that it is important to take those conditions into account when designing and configuring both the end-host and the network mechanisms. 

May I also remind that the Linux Prague code is research driven, and focused only on de-risking the bigger safety challenges. Most of the improvements (except low latency) were not prioritized up to now, as we are sure some are low hanging fruit and others with the right incentives will want to spend time on the harder ones. Most of the commercial parties with an expected interest and benefit of an L4S CC have been reasonable quite, I hope because they are confident that the network drafts and the mandatory Prague requirements and proposed improvements are feasible. I guess some are also less keen on sharing their efforts with competition and others might be waiting for real deployments before getting really into action. If there are concerns from their side, of course we would like to hear those too. The only concern I heard up to now is that all this takes too long before they can start the experiment. They want asap real world deployment...

>> 1) Starting with an easier one, can the reported intolerance to bursts
>> (https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by starting marking in the L queue later? Doing so may also improve RTT fairness for Prague flows.

Indeed, the solution is to make sure that your L4S threshold configuration is in line with your network capabilities. If you know that certain network elements aggregate and burst packets in 4ms chunks, then it makes no sense at all to mark packets in that node at 1ms, nor in subsequent network nodes, with similar bottleneck capacities. We tried to address this by explaining this in section 6.3 of the L4S Architecture draft: https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-08#page-20. Is this sufficient to cover this issue?

>> 2) For the reported network bias, can this be fixed in the DualPI2 and Prague implementations? One chart which we didn't reference in our writeup illustrates the consistent bias for Prague vs CUBIC.

Since an important design goal of L4S is that it also supports network nodes that don't identify individual flows. To compensate individual flow's RTT you need to know them first. To solve this anyways, it would need a per packet indication of the RTT (ref past proposals like RCP and XCP). If we would have space to put this info in the header, we wouldn't be discussing about this single available ECN bit, right? Therefor the End-System is the right place to handle this, and why the RTT-independent requirement is important. The current implementation has a non-Prague-compliant default (not RTT-independent), which we need to fix clearly. It would be good to agree on a reasonable compromise of a default f(RTT) correction function. We also implemented a gradual correction over time, so a new (short) flow would not be delayed, and will be responsive, only if it lasts for a while it will gradually converge to the RTT-independent rate. This could be in the range of the convergence time of the typical-RTT flows.

Feel free to experiment with the available options.

>> * To not consider a 16:1 throughput imbalance between L4S and non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
>> Although we're not sure of the worst case, what we're seeing now is outside of my comfort zone, personally.

Actually, I'm also not happy with this strong RTT dependency. You probably will see similar unfairness cases if you use under these conditions Reno on a CoDel AQM with a target of 5ms. Cubic compensates performance for the longer RTT flows, Prague (with the right settings) compensates mandatory for the smaller RTT. Clearly a production congestion control is likely to do both, or fall back to Classic Cubic behavior whenever the RTT is too long for interactive services.

So the exact out-of-our-confort-zone exists today already, but there are solutions available, just to be integrated.

>> 3) The issues that arise due to the redefinition of CE may be the hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), 
>> and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes)

This is fortunately showing the flaws of CoDel and on top of that the lack of the CoDel implementation to respond with drop when ECN flows get into overload. Even more problematic for Cubic on CoDel (deployed today), your results show it is also bad (more than 2 seconds latency for longer than 10 seconds when throughput reduces from 50Mbps to 1Mbps on an 80ms base RTT path). When compared under the same conditions with Cubic on PI2 you can definitely see how it can be improved using an appropriate AQM (a very short peak of I guess less than 100ms extra latency). Hopefully these CoDel instances can be fixed and can support the ECN threshold for L4S instead (and maybe use PI2 instead of CoDel for Classic). On the other hand these extremes would be also easy cases for both Classic and L4S CCs to detect and fall back on a delay based CC when latencies are that bad (falling back to Classic ECN or Classic drop would even not help to avoid the CoDel problem). But I doubt the Linux Cubic developers would be happy to extend Cubic to avoid CoDel flaws??

>> Although I won't be able to support a WGLC until I see tested code that addresses the issues, I do want to support the WG along its present path to a conclusion...

Try switching on the RTT-independent settings of Prague, and I assume it is a matter of just coding a fallback to Cubic if your detected RTT is too big. So as far as I'm concerned, we do have running code! 

I'll try to get the defaults right. I assume f(RTT)=max(25ms, RTT) and a gradual EWMA convergence reaching the 25ms behavior after about 10 seconds would be good defaults? If people have suggestions or preferences for other values, then let me know. If people have time and want to implement the Cubic fallback (or even better longer RTT-independence), then let me know too.

Regards,
Koen.


-----Original Message-----
From: tsvwg <[e-mail]> On Behalf Of Pete Heist
Sent: Sunday, November 15, 2020 12:42 PM
To: Black, David <[e-mail]>; Sebastian Moeller <[e-mail]>
Cc: tsvwg IETF list <[e-mail]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

Hi guys, and thanks for the replies.

Firstly, there was a new test result added for tunnels yesterday, but I'll put that in a separate thread so it isn't buried.

Beyond that, I think it would be productive to try to figure out what might be done to solve the reported issues, limiting that exploration for now to the confines of the L4S design. We'd like to support forward progress while facing the issues head on if possible. I'll take a shot here as tersely as I can:

1) Starting with an easier one, can the reported intolerance to bursts
(https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by starting marking in the L queue later? Doing so may also improve RTT fairness for Prague flows.

2) For the reported network bias, can this be fixed in the DualPI2 and Prague implementations? One chart which we didn't reference in our writeup illustrates the consistent bias for Prague vs CUBIC.
Essentially, Prague wins until there is an RTT imbalance of 20ms/80ms:
 
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/s1-charts/l4s_network_bias.svg

3) The issues that arise due to the redefinition of CE may be the hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes) 

Afaik, the proposed solutions in the L4S architecture are:

* RFC3168 bottleneck detection in the endpoints. However, we went through a round of testing earlier this year that showed accurate bottleneck detection is likely to be difficult with jitter and cross- flow traffic. It's disabled at present in the reference implementation.

* L4S operational guidance for network operators. So far I haven't had time to review this, but I may have some feedback at least in the area of testing. I suspect the effectiveness of guidance will be influenced by human factors.

* To not consider a 16:1 throughput imbalance between L4S and non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
Although we're not sure of the worst case, what we're seeing now is outside of my comfort zone, personally.


For me, the class of problems in #3 are my area of greatest concern, as there are many more RFC3168 bottlenecks deployed today than just a few years ago (https://github.com/heistp/l4s-tests/#deployments-of-fq_codel) #2 is also important to me, as I trust we're not trying to introduce a so- called "fast lane" for certain traffic.

Although I won't be able to support a WGLC until I see tested code that addresses the issues, I do want to support the WG along its present path to a conclusion...

Pete

On Sun, 2020-11-15 at 01:26 +0000, Black, David wrote:
> Hi Sebastian,
> 
> Some comments as an individual, not a WG chair.
> 
> First, I think Pete has pretty clearly established that TCP Prague is 
> research, and hence (IMHO) TCP Prague ought to be headed for ICCRG as 
> its primary forum rather than TSVWG.
> 
> To date, the 'Prague L4S Requirements' (Appendix A of draft-ietf-
> tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague. That 
> association ought to be teased apart so that the resulting L4S 
> scalable congestion control requirements provide a reasonable design 
> space that can include a number of other congestion control designs - 
> in addition to what's been discussed, e.g., SCReAM, it would be useful 
> to better understand what it would take for implementations of 
> protocols such as DCTCP and BBR (e.g., for QUIC) to meet those 
> requirements.
> 
> My overall take on the requirements is that in 20/20 hindsight, some 
> of them were overly optimistic, and hence need to be backed off/toned 
> down/broadened to encompass what is reasonable in "running code" well 
> beyond TCP Prague. That sort of collision between interesting ideas 
> and network realities is not an unheard-of scenario in IETF, so I 
> hesitate to view the need for changes to these requirements as 
> evidence that the original ideas were inherently defective, as I've 
> seen far more dramatic changes, e.g., some number of years ago, the 
> first design of iSCSI login was elegant ... and resulted in 
> implementations that did not interoperate, resulting in a complete 
> redesign.
> 
> Thanks, --David
> 
> > -----Original Message-----
> > From: tsvwg <[e-mail]> On Behalf Of Sebastian Moeller
> > Sent: Thursday, November 12, 2020 6:11 PM
> > To: Pete Heist
> > Cc: tsvwg IETF list
> > Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow 
> > latency
> > 
> > 
> > [EXTERNAL EMAIL]
> > 
> > Hi Pete,
> > 
> > great data. IMHO Especially the fact that short-RTT Prague will 
> > severely outcompete long-RTT Prague way more than the traditional 
> > dumb FIFO will, strongly supports  the following hypotheses I have 
> > been posting
> > before:
> > 
> > a) Dualpi2/TCP Prague are no way ready for deployment, but are 
> > basically still in the toy stage
> > 
> > b) all that L4S will effectively do, be it by intent or simply by 
> > mis-design, is built a "fast lane" for short RTT low hop count 
> > traffic. Given all the hype
> > (still!) in the L4S
> > internet drafts about how this is the future of the internet...
> > Also a fast lane that
> > requires active cooperation from the leaf ISP (if they keep their 
> > bottlenecks at FIFO, I see no compelling reason for TCP Prague at 
> > all), which will require SLAs between CDNs and ISPs.
> > 
> > c) too little, too late. Really only modest gains in RTT (compared 
> > to best of class AQMs like fq_codel or cake) and noticeable 
> > regressions in sharing behavior both between different CCs and flows 
> > of different RTTs (and that compared to dumb queue "management" with 
> > a simple FIFO).
> > 
> > 
> > 
> > Also quite sobering, that it AGAIN was not team L4S bringing real 
> > data to the table to support their claims and promises. Then again 
> > looking at the RTT fairness for Prague versus Prague under DualQ, I 
> > can understand why team L4S stayed mumm, and rather argued for 
> > allowing unfettered self-mutilation of L4S compliant transport 
> > protocols in regards to RTT fairness:
> > 
> > "So there is no need  to mandate that an L4S implementer does no 
> > harm to themselves, which window-based congestion controls tend to 
> > do at higher RTT.
> > Of course, this doesn't preclude implementers reducing or 
> > eliminating RTT bias for larger than typical RTTs, but it removes 
> > any requirement to do so."
> > 
> > After years of advertising on increased RTT independence (in spite 
> > of the data showing that the proposed combination of DualQ and TCP 
> > Prague actually increases RTT bias), team L4S in the last minute no 
> > less, decides to do a 180 and just change the requirements to allow 
> > for the rather unhealthy behavior demonstrated for L4S...
> > With the current enhanced RTT bias, who in their right mind is going 
> > to use TCP Prague; which currently is the only transport, that at 
> > least attempted to tackle all the "requirements" L4S poses for 
> > transports that want to use the
> > ECT(1) express
> > way? Then again, since none of the network elements are designed and 
> > required to actually check and enforce these requirements, calling 
> > these "requirements" is a bit of stretch anyway, maybe they should 
> > be changed from MUST requirements to COULD musings....
> > 
> > 
> > Best Regards
> >         Sebastian
> > 
> > 
> > > On Nov 12, 2020, at 21:31, Pete Heist <[e-mail]> wrote:
> > > 
> > > Hi,
> > > 
> > > We have posted some new tests of L4S here:
> > > 
> > > https://github.com/heistp/l4s-tests
> > > 
> > > These tests cover two-flow fairness in several qdiscs with 
> > > different path RTTs per-flow, and transient behavior in fq_codel 
> > > queues.
> > > 
> > > The results raise some concerns about a bias in favor of L4S flows 
> > > in
> > > DualPI2 queues, throughput imbalances between flows with different 
> > > path RTTs, and intra-flow latency spikes upon rate reductions in 
> > > fq_codel.
> > > The repo above contains a walk-through of the key findings, and 
> > > links to more results in the Appendix.
> > > 
> > > Regards,
> > > Pete
> > > 
> > > 
>