Re: [tsvwg] Another tunnel/VPN scenario (was RE: Reasons for WGLC/RFC asap)

[Greg White <[e-mail]>]

I'm not sure why I take the bait on these...

Sebastian,

You asked for a realistic scenario where a dramatic reduction in latency variation matters, and I gave you one (well, two).  Just because you don't see the value in improving latency and latency variation doesn't mean there is no value.  There is a strong interest by many in this group (and the broader networking industry) in solutions to Internet latency / latency variation, and high-fidelity congestion signaling is a key and necessary component.  Does it immediately solve all sources of latency? No.  Is TCP Prague (or BBRv2, or SCReAM), as currently implemented, perfect? No.  Is there an issue with RFC3168 interoperability?  Some say yes, some say no.  

But we've heard from a number of very experienced transport protocol and congestion control researchers who see L4S congestion signaling as a feasible path, and we've heard from a large number of network operators who also see it as a feasible path and are willing to make the investment in deploying it.  


Also:

50ms is not the industry target for MTP latency for VR, it is generally 20ms or less.
https://developer.oculus.com/learn/bp-rendering/ 
https://www-file.huawei.com/-/media/CORPORATE/PDF/ilab/cloud_vr_oriented_bearer_network_white_paper_en_v2.pdf 
https://www.cs.unc.edu/~zhengf/ISMAR_2014_Latency.pdf

In isochronous applications, late packets generally *do* mean lost packets.  Current VR gear runs at 90 fps, and cloud games run at up to 120 fps. A 100ms latency spike can mean 9-12 frame times with no data received, and then 9-12 frames arriving back-to-back.  Adaptive dejitter buffers can reduce packet drop rate, but only at the expense of added (and variable) delay.  Either way it is a degradation.


To Wes's point, I'm going to try to leave it there on this thread rather than encouraging even more back and forth.  Don't interpret the lack of further response as agreement.

-Greg



On 12/9/20, 4:08 PM, "Sebastian Moeller" <[e-mail]> wrote:

    Greg,


    > On Dec 9, 2020, at 22:56, Greg White <[e-mail]> wrote:
    > 
    > Sebastian,
    > 
    > As usual, there was a lot of hyperbole and noise in your response, most of which I'll continue to ignore.  But, I will respond to this:

    	[SM] I believe you might not have read that post, what in there is so hyperbolic that it offended your good taste? My predictions of how L4S is going t behave are all extrapolations from data (mostly Pete's, some from team L4S), if you believe these to be wrong, please show how and why.  


    > 
    > 	[SM] Mirja, L4S offer really only very little advancement over the state of the art AQMs, 5ms average queueing delay is current reality, L4S' 1 ms (99.9 quantile) queueing delay really will not change much here, yes 1ms is smaller than 5ms, but please show a realistic  scenario where that difference matters.
    > 
    > 
    > This underscores a perennial misunderstanding of the benefits of L4S.  It isn't about average latency.  Current "state of the art" AQMs result in P99.99 latencies of 40-120 ms (depending on the traffic load),

    	[SM] This is not what I see on my link, sorry. Here are the cake statistics from running 3 concurrent speedtests on three different devices (one of the speedtests exercised 8 bidirectional concurrent flows each marked with one of the CS0 to CS7 diffserv codepoints to exercise all of cake's priority tins).


    qdisc cake 80df: dev pppoe-wan root refcnt 2 bandwidth 31Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 88 
     Sent 1692154603 bytes 4053117 pkt (dropped 2367, overlimits 3194719 requeues 0) 
     backlog 38896b 28p requeues 0
     memory used: 663232b of 4Mb
     capacity estimate: 31Mbit
     min/max network layer size:           28 /    1492
     min/max overhead-adjusted size:       88 /    1542
     average network hdr offset:            0

                       Bulk  Best Effort        Voice
      thresh       1937Kbit       31Mbit     7750Kbit
      target          9.4ms        5.0ms        5.0ms
      interval      104.4ms      100.0ms      100.0ms
      pk_delay       48.8ms       25.0ms       12.5ms
      av_delay        8.1ms        4.7ms        3.7ms
      sp_delay        1.4ms        349us        311us
      backlog         3036b       28348b        7512b
      pkts            48019      3879642       127851
      bytes        19893008   1594898953     80834953
      way_inds            0        93129          167
      way_miss            5        80699          678
      way_cols            0            4            0
      drops               0         2367            0
      marks             262         1322          405
      ack_drop            0            0            0
      sp_flows            1            7            2
      bk_flows            1            8            2
      un_flows            0            0            0
      max_len          1492         1492         1492
      quantum           300          946          300

    qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
     Sent 8865345923 bytes 7688386 pkt (dropped 0, overlimits 0 requeues 0) 
     backlog 0b 0p requeues 0
    qdisc cake 80e0: dev ifb4pppoe-wan root refcnt 2 bandwidth 95Mbit diffserv3 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 88 
     Sent 8862364695 bytes 7686372 pkt (dropped 1971, overlimits 11290382 requeues 0) 
     backlog 64156b 43p requeues 0
     memory used: 567Kb of 4750000b
     capacity estimate: 95Mbit
     min/max network layer size:           28 /    1492
     min/max overhead-adjusted size:       88 /    1542
     average network hdr offset:            0

                       Bulk  Best Effort        Voice
      thresh       5937Kbit       95Mbit    23750Kbit
      target          5.0ms        5.0ms        5.0ms
      interval      100.0ms      100.0ms      100.0ms
      pk_delay        2.0ms        9.2ms        102us
      av_delay        1.1ms        4.5ms         15us
      sp_delay          4us        522us         10us
      backlog            0b       64156b           0b
      pkts            71084      7607860         9442
      bytes       102481496   8762307523       556904
      way_inds            0       223028            0
      way_miss          177        77358          264
      way_cols            0            3            0
      drops              18         1953            0
      marks               0         1014            0
      ack_drop            0            0            0
      sp_flows            1           17            1
      bk_flows            0            8            0
      un_flows            0            0            0
      max_len          1492         1492          638
      quantum           300         1514          724

    Not only does the average delay (av_delay) in the non Bulk (cake's name for the scavenger background class) below 5ms, no even, the peak delay (P100 in your nomenclature) stays well below your predicted 40 to 120ms. These statistics come from an uptime of 8 hours (my ISP reconnects my PPPoE session one per day, at which point the stats get cleared). Now that is an anecdotal data point, but at least it is a real data point. 

    > compared against ~6 ms at P99.99 for L4S.  

    	[SM] I accept your claim, but want to see real data with e.g. real speedtests as artificial load generators over the real existing internet, not simulated runs in a simulated network. 


    > For a cloud-rendered game or VR application, P99.99 packet latency occurs on average once every ~4 seconds (of course, packet latency may not be iid, so in reality high latency events are unlikely to be quite that frequent).  

    	[SM] Well, cloud rendered games are already a commercial reality, so the state of the art can not be so terrible that people are not willing to play.


    > Since RTTs of greater than 20ms have been shown to cause nausea in VR applications, this is a realistic scenario where the difference *really* matters.
    > 
    > Put another way, if, of that 20 ms motion-to-photon budget for VR, a generous 10 ms is allowed for queuing delay (leaving only 10 ms for motion capture, uplink propagation, rendering, compression, downlink propagation, decompression, display), current "state of the art" AQMs would result in 10-50% packet loss (due to late packet arrivals) as opposed to <0.001% with L4S.

    	[SM] Greg, I believe you are trying to paraphrase John Carmack (see for example https://danluu.com/latency-mitigation/, would have been nice if you included a citation). There he also says "A total system latency of 50 milliseconds will feel responsive, but still subtly lagging." And that 50ms goal is well within reach with the state of the art AQMs. But even 20 ms is within reach if the motion tracking is done predictively (think Kalman filters) in which case network delays can be speculated over.

    	But the fact alone that we have to go that deep into the weeds of an obscure example like VR-with-network-rendering to find a single example where L4S's claimed (not realistically proven yet) low queueing delay might be relevant is IMHO telling. Also do you have data showing your claim  "current "state of the art" AQMs would result in 10-50% packet loss (due to late packet arrivals)* as opposed to <0.001% with L4S" is actually true for the existing L4S AQM and transport protocols. To me that reads like a best case prediction and not like hard data. 
             One could call framing predictions as if they were data as "noise" and "hyperbolic", if one would be so inclined... But humor me, and show that these are numbers from real tests over the real internet... 


    IMHO the fact that the best you can offer is this rather contrived VR-in-the-cloud example** basically demonstrates my point that L4S offers only very little over the state of the art, but does so at a considerable cost. 


    Best Regards
    	Sebastian

    *) That is not how packet loss works. A late packet is not a lost packet, and in this example displaying a frame that is 20+Xms in the past is better than not changing the frame at all. 

    **) Which has a very simple solution, don't render VR via the cloud, the won 10ms processing time can be put to good use for e.g. taking later motion data into account for tighter integration between somatosensation and visual inputs... I understand why you resort to this example, but really, this is simply a problem, where cutting out the network completely is the best solution, at least for motion sensor integration.... 

    > 
    > -Greg
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > On 12/4/20, 6:38 AM, "tsvwg on behalf of Sebastian Moeller" <[e-mail] on behalf of [e-mail]> wrote:
    > 
    >    Hi Mirja,
    > 
    >    more below in-line, prefixed with [SM].
    > 
    > 
    >> On Dec 4, 2020, at 13:32, Mirja Kuehlewind <mirja.kuehlewind=[e-mail]> wrote:
    >> 
    >> Hi all,
    >> 
    >> to add one more option.
    >> 
    >> To be clearly any issues discussed only occurred if RCF3168-AQMs (without FQ) are deployed in the network (no matter if any traffic is actually RFC3168-ECN enabled or not). 
    >> 
    >> My understanding of the current deployment status is that only ECN-AQMs with FG support are deployed today and I don't think we should put a lot of effort in a complicated RFC3168-AQM detection mechanism which might negative impact the L4S experiment if we have no evidence that these queues are actually deployed.
    > 
    >    	[SM] So you seem to reject the tunnel argument, Could you please elaborate why tunnels seem ignorable in this context, but are a big argument for re-defining CE? These two positions seem logically hard to bring into alignment. 
    > 
    >> 
    >> Further I would like to note that RCF3168 AQMs are already negatively impacting non ECN traffic and advantaging ECN traffic.
    > 
    >    	[SM] You must mean that rfc3168 enabled flows do not suffer the other-wise required retransmission after a drop and get a slightly faster congestion feed-back? That is a rather small benefit over non-ECN flows, but sure there is a rationale for ECN usage.
    > 
    >> However, I think it's actually a feature of L4S to provide better performance that non-ECN and therefore providing a deployment incentive for L4S,
    > 
    >    	[SM] There is a difference between performing better by doing something better and "making the existing traffic artificially slower" but that is what L4S does (it works on both ends), stacking the deck against non-L4S traffic.
    > 
    >> as long as non-L4S is not starved entirely.
    > 
    >    	[SM] Please define what exactly you consider "starved entirely" to mean otherwise this is not helpful.
    > 
    >> We really, really must stop taking about fairness as equal flow sharing.
    > 
    >    	[SM] Yes, the paper about the "harm" framework that Wes posted earlier seems to be a much better basis here than a simplistic "all flows need to be strictly equal" strawmen criterion.
    > 
    > 
    >> That is not the reality today (e.g. video traffic can take larger shares on low bandwidth links and that keeps it working)
    > 
    >    	[SM] You wish, please talk to end users that want at the same time use concurrent video streaming and jitter sensitive on-line gaming over their internet access link, and the heroic measures they are willing to take to make this work. It is NOT working as desired out of the box in spite of DASH video being adaptive and games requiring very little traffic in comparison. The solution would be to switch video over to CBR type of streaming instead of the current burtsy video delivery (but that is unlikely to change).
    > 
    > 
    >    IMHO L4S will not change much here because a) it still aims to offer rough per flow fairness (at least for flows with similar network paths) and b) the real solution is controlled/targeted un-fairness where a low latency channel is carved out that works in spite of other flows not cooperating (L4S requires implicit coordination/cooperation of all flows to achieve its means, which is to stay civil optimistic).
    > 
    > 
    >> and it is not desired at all because not all flows are equal!!!
    > 
    >    	[SM] Now, if only we had a reliably and robust way to rank flows by importance that is actually available at the bottleneck link we would be set. Not amount of exclamation marks is going to solve that problem, that importance of flows is a badly defined problem. If our flows cross on say our shared IPS's transit uplink, which is more important? And who should be the arbiter of importance, you, me, the ISP, the upstream ISP? That gets complicated quickly, no?
    > 
    > 
    >> The endpoints know what is required to make their application work and as long as there is a circuit breaker that avoids complete starving or collapse, the evolution of the Internet depends on this control in the endpoint and future applications that potentially have very different requirements. Enforcing equal sharing in the network hinders this evolution.
    > 
    >    	[SM] Well, the arguments for equal-ish sharing are:
    >    a) simple to understand, predict and measure/debug (also conceptually easy to implement).
    >    b) avoids starvation as best as possible as evenly as possible
    >    c) is rarely pessimal (and almost never optimal), often "good enough".
    > 
    >    Compare this with your proposed "anything goes" approach (which does not reflect the current internet which seems mostly rough equitable sharing in nature)
    >    a) extremely hard to make predictions unless the end point controls all flows over the bottleneck
    >    b) has not inherent measures against starvation
    >    c) Has the potential to be optimal, but that requires a method to rate relative importance/value of each packet that rarely exist at the points of congestion. 
    > 
    >    How should the routers at a peering point between two AS know, which of the flows in my network I value most? Simply they can't and hence will not come up with the theoretically optimal sharing behavior. I really see no evolutionary argument for anything goes here.
    > 
    >> 
    >> I also would like to note that L4S is not only about lower latency. Latency is the huge problem we have in the Internet today because the current network was optimized for high bandwidth applications, however, many of the critical things we do on the Internet today actually is more sensitive to latency. This problem is still not fully solved, event hough smaller queues and AQM deployments are a good step in the right direction.
    > 
    >    	[SM] Mirja, L4S offer really only very little advancement over the state of the art AQMs, 5ms average queueing delay is current reality, L4S' 1 ms (99.9 quantile) queueing delay really will not change much here, yes 1ms is smaller than 5ms, but please show a realistic  scenario where that difference matters.
    > 
    > 
    >> L4S goes even further and the point is not only about reducing latency but to enable the deployment of a completely new congestion control regime with takes into account all the lessons learnt from e.g. data center deployment where we not have to be bounded by today's limitation of "old" congestion controls and co-existence.
    > 
    >    	[SM] I do smell second system syndrome here. Instead of aiming for a revolution, how about evolving the existing CCs instead? The current attempts at making DCTCP fit for the wider internet in the guise of TCP Prague are quite disappointing in what they actually deliver. To be blunt TCP Prague demonstrates quite well that the initial assumption DCTCP would work well over the internet if only it was safe to do so was simply wrong. The long initial ramp up time and the massively increased RTT-bias as well as the failure to compete well with cubic flows in FIFO bottlenecks are clear indicators that a new L4S reference transport protocol needs to be developed.
    > 
    > 
    >> L4S is exactly a way to transmission to this new regime without starving "old" traffic but there also need to be incentives to actually move to the new world. That's what I would like to see and why I'm existed about L4S.
    > 
    >    	[SM] That is a procedural argument that seems to take L4S's promises at face value, while ignoring all the data that demonstrate L4S has still a long way to go to actually deliver on its promises. 
    >    	I also do not believe it to be an acceptable way to create incentives by essentially making existing transport protocols perform worse (over L4s controlled bottlenecks). But that is what L4S does.
    > 
    >    Best Regards
    >    	Sebastian
    > 
    > 
    >> 
    >> Mirja
    >> 
    >> 
    >> 
    >> 
    >> On 04.12.20, 12:49, "tsvwg on behalf of Michael Welzl" <[e-mail] on behalf of [e-mail]> wrote:
    >> 
    >> 
    >> 
    >>> On Dec 4, 2020, at 12:45 PM, Jonathan Morton <[e-mail]> wrote:
    >>> 
    >>>> On 4 Dec, 2020, at 1:33 pm, Michael Welzl <[e-mail]> wrote:
    >>>> 
    >>>> Right; bad! But the inherent problem is the same: TCP Prague’s inability to detect the 3168-marking AQM algorithm. I thought that a mechanism was added, and then there were discussions of having it or not having it?  Sorry, I didn’t follow this closely enough.
    >>> 
    >>> Right, there was a heuristic added to TCP Prague to (attempt to) detect if the bottleneck was RFC-3168 or L4S.  In the datasets from around March this year, we showed that it didn't work reliably, with both false-positive and false-negative results in a variety of reasonably common scenarios.  This led to both the L4S "benefits" being disabled, and a continuation of the harm to conventional flows, depending on which way the failure went.
    >>> 
    >>> The code is still there but has been disabled by default, so we're effectively back to not having it.  That is reflected in our latest test data.
    >>> 
    >>> I believe the current proposals from L4S are:
    >>> 
    >>> 1: Use the heuristic data in manual network-operations interventions, not automatically.
    >>> 
    >>> 2: Have TCP Prague treat longer-RTT paths as RFC-3168 but shorter ones as L4S.  I assume, charitably, that this would be accompanied by a change in ECT codepoint at origin.
    >>> 
    >>> Those proposals do not seem very convincing to me, but I am just one voice in this WG.
    >> 
    >>   Yeah, so I have added my voice for this particular issue.
    >> 
    >>   Cheers,
    >>   Michael
    >> 
    >> 
    > 
    >