Re: [iccrg] [tsvwg] L4S related activity in 3GPP

Sebastian Moeller <> Sat, 09 November 2019 20:39 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id DE677120020; Sat, 9 Nov 2019 12:39:06 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.349
X-Spam-Status: No, score=-2.349 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id wCxr4AB6bwoI; Sat, 9 Nov 2019 12:39:03 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id D5E22120048; Sat, 9 Nov 2019 12:39:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1573331887; bh=2U/w55u84RuQGmIxjZnIgV2s/qNqxRR9+Q6d0OjEG6U=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=bRjAz3xk2h+KkyNCg4QOVcagEKh44Lu+PFHXZivyQfytyIt8CQJiSfrrc6F3HomKG ATvCdpvIwS9c/J6x4WOVya7tT8A9Z9b/Due4CXH500WYXizMfesJjrMjI54x10Kulv 21ECfDJS7WXRikNMuT8Fayv5wahpiMHa61fdUHT8=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from hms-beagle2.lan ([]) by (mrgmx105 []) with ESMTPSA (Nemesis) id 1MKsj7-1iCqWJ331R-00LDLy; Sat, 09 Nov 2019 21:38:06 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Sat, 9 Nov 2019 21:38:00 +0100
Cc: Ingemar Johansson S <>, "" <>, "" <>, "" <>, "" <>, "Bob Briscoe (" <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <>
To: Ingemar Johansson S <>
X-Mailer: Apple Mail (2.3445.104.11)
X-Provags-ID: V03:K1:60qNxH7vhefk3AKYrGTUAIdVLRpIAdhATok3/4ZzxIwXT9HS76s XuymVyd+jXE+xs6MsM9iisvMNAUZGjzDAyfE+RB59ygP16QIV1A627vP99Tbvc5jsjYLQ7Q sd6nVTXYu7nThs3qDpbRQoKqE/3dsC3C8kRFxc513bJHdV+0w5X5SKQTQF0q8diSPAcc2Qv 9BXHjNQGVB+nAkrvzrBhA==
X-UI-Out-Filterresults: notjunk:1;V03:K0:vaY7kTDS1Fk=:mp2ZNySoQCAUFnd7qay86u FWqHSgr3MO9frIEDHVvJpbHVzgN0doQxwrIUCYu/sAzjb6I6rEeOKNDX4rs2GhHH93OLSVMOl khQBcHAhYSFUAwGfmjn/CnJQorEPq9OJwsfjWPzFz3JBJbD4Z3JNIQHK7LaW54uaEcK0Xl5gf eEo1mDxf9oQSeJ2lVEgq+eKE+OmPj2K7Bf5nx4B7M1V4zQrMQF9OTpxte16Q7NvbdKpY+Uxmp iMOBbAJXJgcpMcks0DpGmd5gfB/dW8hCd19H43TXV+7BaM3RKgdWnLS86oJTFuf14nqVJiAgu kjKJ5PfNAWmvQmtYDon5NhlCu0QBPIuPPMka85lqsIDI8u9yYbMvIYqmXXGoETwtY7WU97EYT +0lmPGO/lc8ySUNVcSfsqczVHdGmoKj5aNLoYqH+mYIPfctu1c/jDbxgunl2V7QicekQCjcRq o8cMDEDHW6Pf9PGi/6SGbKkz5ViH6rzUC/ni8tEPhTo7QoPxnh/7LJslkbtHBqeMJ9ED198GB 3X6tt94En1SDtgjPPtahcRIxYJs+lAphzX+DdYAipUqJBc/p/vClcw7aFQy7qxaZAfnEx2Ea6 6y++UzMTPAwXTyh7Cp0syKRa9z8EMRv75uTRVAnAvgyB0BW3bwDbKuezCoDlZiUcqht/DH4UN 0OlVlEbk1uXC2ZZP+AVwvZPvGtVrLQEIvmeoUBuWM8+wvQ4Hcix6Ze/K5ZKeC4TSGGwtprd+G Lk9pN8VDTvzBgjcsWKlWSWRV1HNRg5gD8G1AvuwiZuzy7OUBAbob//F7ZjdCpRlLfdRL/TWDw pmTqTp4F1x3ju2g1VpbdVKLBwShdFCOs44RD9vSB4aM8gTRkyQYF6G6P5gwpVlDWjZk85uZTf WLN1ja96SAmKF2drOzaISkSkcGQVhzc6376jhmt7EONzw6XYuWYeEdtsEMLGDOYutfUmMtTJk 0wjpb0h4+Q0VriRZmCVvXFHMhKhCUctEfe1K/myRaqh0clKlHaw6aarheExsmBzuTUSnrjs5c +kdWd+KhMSFHDyfYipBowjiOTtmZnbJlnqBrTrlW6Q9TQc7SRupw9LPuZPSAotjfINYnGH0Li 640l5M9Ol5igjySSJtcYdM3zQnWDQvojJrCF+IGnGNag3PDAtUY4QnN+wOaYlv2/WgQZFH4k/ isczskVzyyY/K8WkInuD4xUaejn7yKXoqjvB3o7pybRGpREiHyTtHlovDA8uhhL3dBeLTWGTG LgQZPfxRhjWv2Q1E3DBq+pPfdzUAl9HjccyPtTJRpk0aulJZsgTJrJHo+/4E=
Archived-At: <>
Subject: Re: [iccrg] [tsvwg] L4S related activity in 3GPP
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Discussions of Internet Congestion Control Research Group \(ICCRG\)" <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sat, 09 Nov 2019 20:39:07 -0000

Dear Ingemar,

more in-line below.

> On Nov 9, 2019, at 20:24, Ingemar Johansson S <>; wrote:
> Hi Sebastian 
> Please see inline below
> /Ingemar
>> -----Original Message-----
>> From: Sebastian Moeller <>;
>> Sent: den 9 november 2019 18:13
>> To: Ingemar Johansson S
>> <>;
>> Cc:;;; Ingemar Johansson S
>> <>;;
>> Subject: Re: [tsvwg] L4S related activity in 3GPP
>> Dear Ingemar,
>> thanks for posting this. I read "This document describes an Active Queue
>> Management mechanism, ‘L4S’, that can support low-latency services in a way
>> that does not degrade the performance of other traffic." and I wonder whether
>> 3GPP has run any experiments to confirm/support this claim, that you are willing
>> to share?
> [IJ] We have done simulation studies with our proprietary 4G/5G system simulation tools but I can not disclose the results from these.

	[SM] Ah, I see, not posting internal data is fine, but by all means, go and perform real-world measurements as well with all the parts you actually want to roll-out (in other words testing with dctcp for example is scientifically interesting, unless you expect your users to actually use dctcp on your links).

>> Sidenote: in the powerpoint file it says about L4s " Standardized in IETF [1]",
>> which seems premature, in the process of being standardized seems a better
>> description of the status quo...
> [IJ] Yes, you are right about this, it should have been "Subject to standardization..."

	[SM] Fair enough, as much as I regret to say it, standardization is quite likely.

>> I ask about test data from your side as the recent test data is quite sobering
>> from a number of angles, including quality of isolation:
> [IJ] First of all the outline of the proposal for L4S in 5G is to use separate radio bearers for L4S and classic (or normal or non-L4S) traffic. This means that it is the 5G QoS framework and the scheduling in the RAN (Radio Access Network) that dictates the fairness.

	[SM] Sounds like a workable approach, that side steps the dualQ implementation/theory issues.

> But of course there are wireline transport link in the core network that may be using DualQ when that becomes commercially available. The bitrates over these links are however likely to be considerably higher than 50Mbps.

	That does not matter, the issue is not the limit of 50 Mbps, but the fact that dualQ, at last the current implementation in conjunction with the current TCP prague implementation, simply do not reliably and robustly do what the rfc draft claims. My fear is that dualQ, with its weak coupling vie marking probabilities, simply is a design, that only guarantees approximate fairness. In other words I doubt that dualQ works as a naive reading of the RFC implies. But if I  understood Bob correct, the only gurantee he is willing to give is L4S does not starve non-L4S traffic. Which IMHO is not sufficient to merit inflicting the L4S experiment on the wider internet. Now, that is not my call to make, but the IETF process allows me t voice my concerns, and I just use this opportunity in an attempt to minimize undesired side-effects on the "normal" internet.

>> 	Looking at the current L4S tests from the various testbeds I see severe
>> breakdown of the promised isolation between bulk and the "low-latency" queue,
>> apparently making the current implementation, if not the concept itself IMHO
>> unfit for release into the wider internet, before these issues have been
>> sufficiently addressed. Especially the observation that the weighted scheduler
>> built in as a safety mechanism in the dualQ aqm seems to trigger considerably
>> more often than expected/predicted from reading the dualQ draft (giving 1/16
>> of the bandwidth to on of two classes _might_ be justified as a temporary stop-
>> gap measure in extremely unlikely cases, but this already triggers in a simple
>> one-flow-per queue scenario at low RTTs). In short it currently looks like the
>> "does not degrade performance of other traffic" part is not yet finished.
>> 	To illustrate my criticism, here is a link to a test comparing the simplest
>> functionality test imaginable, running one L4S-style and one non-L4S-style TCP
>> connection through the L4S dual queue AQM at a intranet like RTT (nomnally
>> 0ms, but in reality):
>> 0cc47ad93e1c-6acfac650abe7f09&q=1&e=6cd6c372-b712-46be-8b10-
>> 22955dec7b38&
>> testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.png
>> As can be seen the L4S flow (labeled prague) ends up very quickly taking ~90% of
>> the available bandwidth, leaving only 10% to the non-L4S flow (labeled cubic),
>> wit increasing RTT this bias gets less extreme, but even at 80ms the prague flow
>> still gets around 1.5 times the bandwidth of the cubic flow
>> (
>> 0cc47ad93e1c-56d0590db27309ef&q=1&e=6cd6c372-b712-46be-8b10-
>> 22955dec7b38&
>> testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-80ms_var.png),
>> indicating an RTT dependent performance degradation for "other traffic". Test
>> from another testbed confirm the break-down of fairness between L4S and non-
>> L4S traffic at short RTTs (
>> 0b995f26-5710b074-0cc47ad93e1c-418920d155a62bf0&q=1&e=6cd6c372-
>> b712-46be-8b10-
>> 22955dec7b38&
>> bakeoff%2Fbakeoff-2019-09-13T045427-r1%2Fl4s-s1-2%2Fbatch-l4s-s1-2-
>> cubic-vs-prague-50Mbit-0ms_fixed.png).
> [IJ] I understand that the 0 RTT results can be because the congestion window is only a few segments large (my guess is ~3 to 4 segments ?). I know that this is a problem with e.g. DCTCP.  The question is whether this is a permanent problem or something that will be solved over time?.

	[SM] With DualQ, I assume it is a structural problem, instead of designing this as a robust system with two fair sharing queues it does its clever "indirect coupling" via marking/dropping probabilities. Have a look at for how sharing in a FQ system looks. Even with the leaky ECT(1) as classifier abuse it shoukd be simple to fair queue between EXT(1) and not ECH(1) flows and have robust separation. This is not rocket science, and mostly, as far as I can tell a solved problem.

> What is more interesting to me and what makes L4S attractive for the use cases outlined in the 3GPP contributions, is the very low queue delay with L4S.

	[SM] Well, I believe low queueing delay can be achieved without dualQ and without abusing the ECT(1) codepoint as a classifier (the desired classifier requires one complete independent bit, ECT(1) is just 1/2 of a bit, and hence has failure modes in separating L4S-style flows and non-L4S flows). The idea of using the finer-grained congestion signaling of DCTCP over the wider internet seems sane see the SCE draft proposal as an alternative that would/will allow dctcp-style ECN signaling over the internet in a truly backward compatible fashion...

> There are other promising features with L4S. One being that it can enable congestion control algorithms to quickly grab free capacity.

	[SM] When you say L4S, do you mean the congestion signaling? Then should have you covered, without the nasty side-effects of the current L4S proposal. Or do you mean the actual congestion controller as implemented in TCP prague? 

> 5G access can have large variations in link throughput for example related to dual connectivity and L4S can be helpful here as well.

	[SM] Dealing better with variable rate links is certainly desirable, but I am not sure that DualQ/TCP Prague really are the best or only tools for that purpose. At least not until all of the kinks are ironed out. Again, none of this is my call to make, so all I am voicing is my educated subjective assessments.

Best Regards

>> 	Personally, I can not help thinking that it might be better to first make
>> sure there is a complete, real-world tested implementation that reliably
>> demonstrates that dualQ delivers the required isolation between the two traffic
>> classes and that the L4S approach actually delivers the reported reliable low
>> latency performance under adverse real-world conditions while not degrading
>> the performance of other traffic, before trying to codify L4S into RFCs.
>> Best Regards
>> 	Sebastian
>>> On Nov 9, 2019, at 17:00, Ingemar Johansson S
>> <>; wrote:
>>> Gentlepeople!
>>> This 3GPP SA2 contribution is recently submitted
>>> ad93e1c-77a60175f8460bc6&q=1&e=6cd6c372-b712-46be-8b10-
>> 22955dec7b38&u=
>> _Reno%
>>> Title “5QI values for efficient handling of
>>> congestion control marked IP packets”
>>> The contribution introduces L4S to the SA2 audience and explains where the
>> technology is useful. The contribution also proposes a method to bake L4S into
>> the 5G QoS framework.
>>> If agenda time permits, it will be discussed at the SA WG2 meeting in Reno on
>> Nov 18-22, hopefully with good results.
>>> Also a RAN2 contribution was submitted and discussed at an earlier RAN2
>> meeting.
>>> ad93e1c-f387514c9f633599&q=1&e=6cd6c372-b712-46be-8b10-
>> 22955dec7b38&u=
>> 07bis%2F
>>> Enjoy
>>> /Ingemar
>>> ================================
>>> Ingemar Johansson  M.Sc.
>>> Master Researcher
>>> Ericsson Research
>>> Network Protocols & E2E Performance
>>> Labratoriegränd 11
>>> 971 28, Luleå, Sweden
>>> Phone +46-1071 43042
>>> SMS/MMS +46-73 078 3289
>>>      The best way to find out if you
>>>  can trust somebody is to trust them
>>>              Ernest Hemingway
>>> =================================