Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Sebastian Moeller <moeller0@gmx.de>
In-Reply-To: <HE1PR07MB4425ECE1C3883C6D200AE3A3C2750@HE1PR07MB4425.eurprd07.prod.outlook.com>
Date: Sun, 10 Nov 2019 11:50:27 +0100
Cc: Ingemar Johansson S <ingemar.s.johansson=40ericsson.com@dmarc.ietf.org>,
 "tsvwg@ietf.org" <tsvwg@ietf.org>, "tcpm@ietf.org" <tcpm@ietf.org>,
 "iccrg@irtf.org" <iccrg@irtf.org>,
 "koen.de_schepper@nokia.com" <koen.de_schepper@nokia.com>,
 "Bob Briscoe (ietf@bobbriscoe.net)" <ietf@bobbriscoe.net>
Content-Transfer-Encoding: quoted-printable
Message-Id: <4B7BF37B-18DB-429F-9B7E-1D87629B40C1@gmx.de>
References: <HE1PR07MB4425A148B5BB1E6FD8E5A3FBC27A0@HE1PR07MB4425.eurprd07.prod.outlook.com>
 <2083E62F-0E6D-40B1-B726-A198BFA36220@gmx.de>
 <HE1PR07MB4425DD6FE15DB130B24BCEA1C27A0@HE1PR07MB4425.eurprd07.prod.outlook.com>
 <DBE43199-0BFB-464B-8045-919C975397AC@gmx.de>
 <HE1PR07MB4425ECE1C3883C6D200AE3A3C2750@HE1PR07MB4425.eurprd07.prod.outlook.com>
To: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/Pkia1zZIg6j6q0-qxCkIHfZ4nKk>
Subject: Re: [tsvwg] L4S related activity in 3GPP
Precedence: list

Hi Ingemar,


> On Nov 10, 2019, at 11:20, Ingemar Johansson S =
<ingemar.s.johansson@ericsson.com> wrote:
>=20
> Hi Sebastian
> Please see inline
>=20
> /Ingemar
>=20
>> -----Original Message-----
>> From: Sebastian Moeller <moeller0@gmx.de>
>> Sent: den 9 november 2019 21:38
>> To: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
>> Cc: Ingemar Johansson S
>> <ingemar.s.johansson=3D40ericsson.com@dmarc.ietf.org>; =
tsvwg@ietf.org;
>> tcpm@ietf.org; iccrg@irtf.org; koen.de_schepper@nokia.com; Bob =
Briscoe
>> (ietf@bobbriscoe.net) <ietf@bobbriscoe.net>
>> Subject: Re: [tsvwg] L4S related activity in 3GPP
>>=20
>> Dear Ingemar,
>>=20
>> more in-line below.
>>=20
>>> On Nov 9, 2019, at 20:24, Ingemar Johansson S
>> <ingemar.s.johansson@ericsson.com> wrote:
>>>=20
>>> Hi Sebastian
>>>=20
>>> Please see inline below
>>> /Ingemar
>>>=20
>>>> -----Original Message-----
>>>> From: Sebastian Moeller <moeller0@gmx.de>
>>>> Sent: den 9 november 2019 18:13
>>>> To: Ingemar Johansson S
>>>> <ingemar.s.johansson=3D40ericsson.com@dmarc.ietf.org>
>>>> Cc: tsvwg@ietf.org; tcpm@ietf.org; iccrg@irtf.org; Ingemar =
Johansson
>>>> S <ingemar.s.johansson@ericsson.com>; koen.de_schepper@nokia.com
>>>> Subject: Re: [tsvwg] L4S related activity in 3GPP
>>>>=20
>>>> Dear Ingemar,
>>>>=20
>>>> thanks for posting this. I read "This document describes an Active
>>>> Queue Management mechanism, =E2=80=98L4S=E2=80=99, that can support =
low-latency
>>>> services in a way that does not degrade the performance of other
>>>> traffic." and I wonder whether 3GPP has run any experiments to
>>>> confirm/support this claim, that you are willing to share?
>>>=20
>>> [IJ] We have done simulation studies with our proprietary 4G/5G =
system
>> simulation tools but I can not disclose the results from these.
>>=20
>> 	[SM] Ah, I see, not posting internal data is fine, but by all =
means, go and
>> perform real-world measurements as well with all the parts you =
actually want to
>> roll-out (in other words testing with dctcp for example is =
scientifically
>> interesting, unless you expect your users to actually use dctcp on =
your links).
>=20
> [IJ] I hope to have data to show later on

	[SM] Excellent, looking forward to that.

>=20
>>=20
>>=20
>>>>=20
>>>> Sidenote: in the powerpoint file it says about L4s " Standardized =
in
>>>> IETF [1]", which seems premature, in the process of being
>>>> standardized seems a better description of the status quo...
>>>=20
>>> [IJ] Yes, you are right about this, it should have been "Subject to
>> standardization..."
>>=20
>> 	[SM] Fair enough, as much as I regret to say it, standardization =
is quite
>> likely.
>>=20
>>>=20
>>>>=20
>>>>=20
>>>> I ask about test data from your side as the recent test data is =
quite
>>>> sobering from a number of angles, including quality of isolation:
>>>=20
>>> [IJ] First of all the outline of the proposal for L4S in 5G is to =
use separate radio
>> bearers for L4S and classic (or normal or non-L4S) traffic. This =
means that it is
>> the 5G QoS framework and the scheduling in the RAN (Radio Access =
Network)
>> that dictates the fairness.
>>=20
>> 	[SM] Sounds like a workable approach, that side steps the dualQ
>> implementation/theory issues.
>=20
> [IJ] The proposed approach for 5G is for practical reasons, we believe =
that it is more implementation and standardization friendly as it will =
utilize the existing QoS framework as much as possible.=20

	[SM] Well, what ever the rationale, by avoiding the =
insufficiently isolating DualQ AQM you will save yourself from a number =
of headaches; I simply assume that you implement stricter airtime =
sharing between the two bearers than dualQ, and if you err you err on =
the side of conserving normal traffic (unlike DualQ's "L4S first" =
approach). But you still are saddled with L4S poor choice of ECT(1) as =
heuristic to separate L4S from non-L4S traffic. Calling that aETC(1) =
classifier seems plain wrong, as classifier implies strict separation.


Best Regards
	Sebastian


>=20
>>=20
>>=20
>>> But of course there are wireline transport link in the core network =
that may be
>> using DualQ when that becomes commercially available. The bitrates =
over these
>> links are however likely to be considerably higher than 50Mbps.
>>=20
>> 	That does not matter, the issue is not the limit of 50 Mbps, but =
the fact
>> that dualQ, at last the current implementation in conjunction with =
the current
>> TCP prague implementation, simply do not reliably and robustly do =
what the rfc
>> draft claims. My fear is that dualQ, with its weak coupling vie =
marking
>> probabilities, simply is a design, that only guarantees approximate =
fairness. In
>> other words I doubt that dualQ works as a naive reading of the RFC =
implies. But
>> if I  understood Bob correct, the only gurantee he is willing to give =
is L4S does
>> not starve non-L4S traffic. Which IMHO is not sufficient to merit =
inflicting the
>> L4S experiment on the wider internet. Now, that is not my call to =
make, but the
>> IETF process allows me t voice my concerns, and I just use this =
opportunity in an
>> attempt to minimize undesired side-effects on the "normal" internet.
>>=20
>>>=20
>>>>=20
>>>> 	Looking at the current L4S tests from the various testbeds I see
>>>> severe breakdown of the promised isolation between bulk and the
>>>> "low-latency" queue, apparently making the current implementation, =
if
>>>> not the concept itself IMHO unfit for release into the wider
>>>> internet, before these issues have been sufficiently addressed.
>>>> Especially the observation that the weighted scheduler built in as =
a
>>>> safety mechanism in the dualQ aqm seems to trigger considerably =
more
>>>> often than expected/predicted from reading the dualQ draft (giving
>>>> 1/16 of the bandwidth to on of two classes _might_ be justified as =
a
>>>> temporary stop- gap measure in extremely unlikely cases, but this
>>>> already triggers in a simple one-flow-per queue scenario at low =
RTTs). In
>> short it currently looks like the "does not degrade performance of =
other traffic"
>> part is not yet finished.
>>>>=20
>>>> 	To illustrate my criticism, here is a link to a test comparing =
the
>>>> simplest functionality test imaginable, running one L4S-style and =
one
>>>> non-L4S-style TCP connection through the L4S dual queue AQM at a
>>>> intranet like RTT (nomnally 0ms, but in reality):
>>>> https://protect2.fireeye.com/v1/url?k=3Dfcd5c73d-a05c68f4-fcd587a6-
>>>> 0cc47ad93e1c-6acfac650abe7f09&q=3D1&e=3D6cd6c372-b712-46be-8b10-
>>>> 22955dec7b38&u=3Dhttps%3A%2F%2Fl4s.cablelabs.com%2Fl4s-
>>>> =
testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.p
>>>> ng As can be seen the L4S flow (labeled prague) ends up very =
quickly
>>>> taking ~90% of the available bandwidth, leaving only 10% to the
>>>> non-L4S flow (labeled cubic), wit increasing RTT this bias gets =
less
>>>> extreme, but even at 80ms the prague flow still gets around 1.5 =
times
>>>> the bandwidth of the cubic flow
>>>> (https://protect2.fireeye.com/v1/url?k=3D77272802-2bae87cb-77276899-
>>>> 0cc47ad93e1c-56d0590db27309ef&q=3D1&e=3D6cd6c372-b712-46be-8b10-
>>>> 22955dec7b38&u=3Dhttps%3A%2F%2Fl4s.cablelabs.com%2Fl4s-
>>>> =
testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-80ms_var.
>>>> png), indicating an RTT dependent performance degradation for =
"other
>>>> traffic". Test from another testbed confirm the break-down of
>>>> fairness between L4S and non- L4S traffic at short RTTs
>>>> (https://protect2.fireeye.com/v1/url?k=3D5710f0ef-
>>>> 0b995f26-5710b074-0cc47ad93e1c-418920d155a62bf0&q=3D1&e=3D6cd6c372-
>>>> b712-46be-8b10-
>>>> 22955dec7b38&u=3Dhttps%3A%2F%2Fwww.heistp.net%2Fdownloads%2Fsce-
>> l4s-
>>>> bakeoff%2Fbakeoff-2019-09-13T045427-r1%2Fl4s-s1-2%2Fbatch-l4s-s1-2-
>>>> cubic-vs-prague-50Mbit-0ms_fixed.png).
>>>=20
>>> [IJ] I understand that the 0 RTT results can be because the =
congestion window
>> is only a few segments large (my guess is ~3 to 4 segments ?). I know =
that this is
>> a problem with e.g. DCTCP.  The question is whether this is a =
permanent
>> problem or something that will be solved over time?.
>>=20
>> 	[SM] With DualQ, I assume it is a structural problem, instead of
>> designing this as a robust system with two fair sharing queues it =
does its clever
>> "indirect coupling" via marking/dropping probabilities. Have a look =
at
>> https://protect2.fireeye.com/v1/url?k=3D53bd001b-0f37caa5-53bd4080-
>> 869a17b5b21b-e4edb2934547a534&q=3D1&e=3D314d9ef9-5cc0-4c98-a599-
>> 59f030e6cb2f&u=3Dhttps%3A%2F%2Fwww.heistp.net%2Fdownloads%2Fsce-l4s-
>> bakeoff%2Fbakeoff-2019-09-13T045427-r1%2Fl4s-s2-2%2Fbatch-l4s-s2-2-
>> cubic-vs-prague-50Mbit-10ms_fixed.png for how sharing in a FQ system =
looks.
>> Even with the leaky ECT(1) as classifier abuse it shoukd be simple to =
fair queue
>> between EXT(1) and not ECH(1) flows and have robust separation. This =
is not
>> rocket science, and mostly, as far as I can tell a solved problem.
>>=20
>>> What is more interesting to me and what makes L4S attractive for the =
use
>> cases outlined in the 3GPP contributions, is the very low queue delay =
with L4S.
>>=20
>> 	[SM] Well, I believe low queueing delay can be achieved without =
dualQ
>> and without abusing the ECT(1) codepoint as a classifier (the desired =
classifier
>> requires one complete independent bit, ECT(1) is just 1/2 of a bit, =
and hence has
>> failure modes in separating L4S-style flows and non-L4S flows). The =
idea of using
>> the finer-grained congestion signaling of DCTCP over the wider =
internet seems
>> sane see the SCE draft proposal as an alternative that would/will =
allow dctcp-
>> style ECN signaling over the internet in a truly backward compatible =
fashion...
>>=20
>>=20
>>> There are other promising features with L4S. One being that it can =
enable
>> congestion control algorithms to quickly grab free capacity.
>>=20
>> 	[SM] When you say L4S, do you mean the congestion signaling? =
Then
>> https://tools.ietf.org/html/draft-morton-taht-sce-00 should have you =
covered,
>> without the nasty side-effects of the current L4S proposal. Or do you =
mean the
>> actual congestion controller as implemented in TCP prague?
>>=20
>>=20
>>> 5G access can have large variations in link throughput for example =
related to
>> dual connectivity and L4S can be helpful here as well.
>>=20
>> 	[SM] Dealing better with variable rate links is certainly =
desirable, but I
>> am not sure that DualQ/TCP Prague really are the best or only tools =
for that
>> purpose. At least not until all of the kinks are ironed out. Again, =
none of this is
>> my call to make, so all I am voicing is my educated subjective =
assessments.
>>=20
>>=20
>> Best Regards
>> 	Sebastian
>>=20
>>>=20
>>>>=20
>>>> 	Personally, I can not help thinking that it might be better to =
first
>>>> make sure there is a complete, real-world tested implementation =
that
>>>> reliably demonstrates that dualQ delivers the required isolation
>>>> between the two traffic classes and that the L4S approach actually
>>>> delivers the reported reliable low latency performance under =
adverse
>>>> real-world conditions while not degrading the performance of other =
traffic,
>> before trying to codify L4S into RFCs.
>>>>=20
>>>>=20
>>>> Best Regards
>>>> 	Sebastian
>>>>=20
>>>>=20
>>>>> On Nov 9, 2019, at 17:00, Ingemar Johansson S
>>>> <ingemar.s.johansson=3D40ericsson.com@dmarc.ietf.org> wrote:
>>>>>=20
>>>>> Gentlepeople!
>>>>>=20
>>>>> This 3GPP SA2 contribution is recently submitted
>>>>> =
https://protect2.fireeye.com/v1/url?k=3D803d870e-dcb428c7-803dc795-0cc
>>>>> 47
>>>>> ad93e1c-77a60175f8460bc6&q=3D1&e=3D6cd6c372-b712-46be-8b10-
>>>> 22955dec7b38&u=3D
>>>>>=20
>>>>=20
>> https%3A%2F%2Fwww.3gpp.org%2Fftp%2Ftsg_sa%2FWG2_Arch%2FTSGS2_136
>>>> _Reno%
>>>>> 2FDocs%2FS2-1911250.zip Title =E2=80=9C5QI values for efficient =
handling of
>>>>> congestion control marked IP packets=E2=80=9D
>>>>> The contribution introduces L4S to the SA2 audience and explains
>>>>> where the
>>>> technology is useful. The contribution also proposes a method to =
bake
>>>> L4S into the 5G QoS framework.
>>>>> If agenda time permits, it will be discussed at the SA WG2 meeting
>>>>> in Reno on
>>>> Nov 18-22, hopefully with good results.
>>>>>=20
>>>>> Also a RAN2 contribution was submitted and discussed at an earlier
>>>>> RAN2
>>>> meeting.
>>>>> =
https://protect2.fireeye.com/v1/url?k=3D06eb4ef7-5a62e13e-06eb0e6c-0cc
>>>>> 47
>>>>> ad93e1c-f387514c9f633599&q=3D1&e=3D6cd6c372-b712-46be-8b10-
>>>> 22955dec7b38&u=3D
>>>>>=20
>>>>=20
>> https%3A%2F%2Fwww.3gpp.org%2Fftp%2FTSG_RAN%2FWG2_RL2%2FTSGR2_1
>>>> 07bis%2F
>>>>> Docs%2FR2-1913888.zip
>>>>>=20
>>>>> Enjoy
>>>>> /Ingemar
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>> Ingemar Johansson  M.Sc.
>>>>> Master Researcher
>>>>>=20
>>>>> Ericsson Research
>>>>> Network Protocols & E2E Performance
>>>>> Labratoriegr=C3=A4nd 11
>>>>> 971 28, Lule=C3=A5, Sweden
>>>>> Phone +46-1071 43042
>>>>> SMS/MMS +46-73 078 3289
>>>>> ingemar.s.johansson@ericsson.com
>>>>> www.ericsson.com
>>>>>=20
>>>>>     The best way to find out if you  can trust somebody is to =
trust
>>>>> them
>>>>>             Ernest Hemingway
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D