Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

[Pete Heist <[e-mail]>]

Hi guys, and thanks for the replies.

Firstly, there was a new test result added for tunnels yesterday, but
I'll put that in a separate thread so it isn't buried.

Beyond that, I think it would be productive to try to figure out what
might be done to solve the reported issues, limiting that exploration
for now to the confines of the L4S design. We'd like to support forward
progress while facing the issues head on if possible. I'll take a shot
here as tersely as I can:

1) Starting with an easier one, can the reported intolerance to bursts
(https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by
starting marking in the L queue later? Doing so may also improve RTT
fairness for Prague flows.

2) For the reported network bias, can this be fixed in the DualPI2 and
Prague implementations? One chart which we didn't reference in our
writeup illustrates the consistent bias for Prague vs CUBIC.
Essentially, Prague wins until there is an RTT imbalance of 20ms/80ms:
 
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/s1-charts/l4s_network_bias.svg

3) The issues that arise due to the redefinition of CE may be the
hardest. This includes the domination of L4S over non-L4S flows in the
same RFC3168 queue (see the issue with tunnels reported above for a
common path to that), and the intra-flow latency spikes for L4S flows
(https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes) 

Afaik, the proposed solutions in the L4S architecture are:

* RFC3168 bottleneck detection in the endpoints. However, we went
through a round of testing earlier this year that showed accurate
bottleneck detection is likely to be difficult with jitter and cross-
flow traffic. It's disabled at present in the reference implementation.

* L4S operational guidance for network operators. So far I haven't had
time to review this, but I may have some feedback at least in the area
of testing. I suspect the effectiveness of guidance will be influenced
by human factors.

* To not consider a 16:1 throughput imbalance between L4S and non-L4S
flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
Although we're not sure of the worst case, what we're seeing now is
outside of my comfort zone, personally.


For me, the class of problems in #3 are my area of greatest concern, as
there are many more RFC3168 bottlenecks deployed today than just a few
years ago
(https://github.com/heistp/l4s-tests/#deployments-of-fq_codel) #2 is
also important to me, as I trust we're not trying to introduce a so-
called "fast lane" for certain traffic.

Although I won't be able to support a WGLC until I see tested code that
addresses the issues, I do want to support the WG along its present
path to a conclusion...

Pete

On Sun, 2020-11-15 at 01:26 +0000, Black, David wrote:
> Hi Sebastian,
> 
> Some comments as an individual, not a WG chair.
> 
> First, I think Pete has pretty clearly established that TCP Prague is
> research, and hence (IMHO) TCP Prague ought to be headed for ICCRG as
> its primary forum rather than TSVWG.
> 
> To date, the 'Prague L4S Requirements' (Appendix A of draft-ietf-
> tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague. 
> That association ought to be teased apart so that the resulting L4S
> scalable congestion control requirements provide a reasonable design
> space that can include a number of other congestion control designs -
> in addition to what's been discussed, e.g., SCReAM, it would be
> useful to better understand what it would take for implementations of
> protocols such as DCTCP and BBR (e.g., for QUIC) to meet those
> requirements.
> 
> My overall take on the requirements is that in 20/20 hindsight, some
> of them were overly optimistic, and hence need to be backed off/toned
> down/broadened to encompass what is reasonable in "running code" well
> beyond TCP Prague. That sort of collision between interesting ideas
> and network realities is not an unheard-of scenario in IETF, so I
> hesitate to view the need for changes to these requirements as
> evidence that the original ideas were inherently defective, as I've
> seen far more dramatic changes, e.g., some number of years ago, the
> first design of iSCSI login was elegant ... and resulted in
> implementations that did not interoperate, resulting in a complete
> redesign.
> 
> Thanks, --David
> 
> > -----Original Message-----
> > From: tsvwg <[e-mail]> On Behalf Of Sebastian Moeller
> > Sent: Thursday, November 12, 2020 6:11 PM
> > To: Pete Heist
> > Cc: tsvwg IETF list
> > Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow
> > latency
> > 
> > 
> > [EXTERNAL EMAIL]
> > 
> > Hi Pete,
> > 
> > great data. IMHO Especially the fact that short-RTT Prague will
> > severely
> > outcompete long-RTT Prague way more than the traditional dumb FIFO
> > will,
> > strongly supports  the following hypotheses I have been posting
> > before:
> > 
> > a) Dualpi2/TCP Prague are no way ready for deployment, but are
> > basically still in
> > the toy stage
> > 
> > b) all that L4S will effectively do, be it by intent or simply by
> > mis-design, is built a
> > "fast lane" for short RTT low hop count traffic. Given all the hype
> > (still!) in the L4S
> > internet drafts about how this is the future of the internet...
> > Also a fast lane that
> > requires active cooperation from the leaf ISP (if they keep their
> > bottlenecks at
> > FIFO, I see no compelling reason for TCP Prague at all), which will
> > require SLAs
> > between CDNs and ISPs.
> > 
> > c) too little, too late. Really only modest gains in RTT (compared
> > to best of class
> > AQMs like fq_codel or cake) and noticeable regressions in sharing
> > behavior both
> > between different CCs and flows of different RTTs (and that
> > compared to dumb
> > queue "management" with a simple FIFO).
> > 
> > 
> > 
> > Also quite sobering, that it AGAIN was not team L4S bringing real
> > data to the table
> > to support their claims and promises. Then again looking at the RTT
> > fairness for
> > Prague versus Prague under DualQ, I can understand why team L4S
> > stayed
> > mumm, and rather argued for allowing unfettered self-mutilation of
> > L4S
> > compliant transport protocols in regards to RTT fairness:
> > 
> > "So there is no need  to mandate that an L4S implementer does no
> > harm to
> > themselves, which window-based congestion controls tend to do at
> > higher RTT.
> > Of course, this doesn't preclude implementers reducing or
> > eliminating RTT bias
> > for larger than typical RTTs, but it removes any requirement to do
> > so."
> > 
> > After years of advertising on increased RTT independence (in spite
> > of the data
> > showing that the proposed combination of DualQ and TCP Prague
> > actually
> > increases RTT bias), team L4S in the last minute no less, decides
> > to do a 180 and
> > just change the requirements to allow for the rather unhealthy
> > behavior
> > demonstrated for L4S...
> > With the current enhanced RTT bias, who in their right mind is
> > going to use TCP
> > Prague; which currently is the only transport, that at least
> > attempted to tackle all
> > the "requirements" L4S poses for transports that want to use the
> > ECT(1) express
> > way? Then again, since none of the network elements are designed
> > and required
> > to actually check and enforce these requirements, calling these
> > "requirements" is
> > a bit of stretch anyway, maybe they should be changed from MUST
> > requirements
> > to COULD musings....
> > 
> > 
> > Best Regards
> >         Sebastian
> > 
> > 
> > > On Nov 12, 2020, at 21:31, Pete Heist <[e-mail]> wrote:
> > > 
> > > Hi,
> > > 
> > > We have posted some new tests of L4S here:
> > > 
> > > https://github.com/heistp/l4s-tests
> > > 
> > > These tests cover two-flow fairness in several qdiscs with
> > > different
> > > path RTTs per-flow, and transient behavior in fq_codel queues.
> > > 
> > > The results raise some concerns about a bias in favor of L4S
> > > flows in
> > > DualPI2 queues, throughput imbalances between flows with
> > > different
> > > path RTTs, and intra-flow latency spikes upon rate reductions in
> > > fq_codel.
> > > The repo above contains a walk-through of the key findings, and
> > > links
> > > to more results in the Appendix.
> > > 
> > > Regards,
> > > Pete
> > > 
> > > 
>