Re: [tsvwg] path forward on L4S issue #16

Sebastian Moeller <> Wed, 17 June 2020 14:16 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 16DCF3A085A; Wed, 17 Jun 2020 07:16:19 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.649
X-Spam-Status: No, score=-1.649 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id tq1WfB9GSolT; Wed, 17 Jun 2020 07:16:16 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 7038E3A0863; Wed, 17 Jun 2020 07:16:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1592403368; bh=KknWDLWyNewAP9dTK1P+KRkhow+ixMLmkoLsTOK8WFk=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=lAIPpal7/vOYsW25Xqn3NCurtaUOEzuku4DMbI29jvH8YR1nPa73aOS0nTZ9XsSbd m4ZH2XXigl162qJ43mRHLW31G1W28dbbkhdXLrkJ0zBYNGEckcrR9hcPc2ahwxHo7x cJp4FRbCvMmU68RRt+xa81V0qRUVnbTOJpp/0ink=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (mrgmx104 []) with ESMTPSA (Nemesis) id 1MSc1B-1jJHQP1EXx-00Su1o; Wed, 17 Jun 2020 16:16:08 +0200
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.14\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Wed, 17 Jun 2020 16:16:05 +0200
Cc: Ingemar Johansson S <>, "Holland, Jake" <>, Jonathan Morton <>, "" <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <>
To: Ingemar Johansson S <>
X-Mailer: Apple Mail (2.3445.104.14)
X-Provags-ID: V03:K1:jwVjwwvgDUaAhx1RnUcaLBHhcj4Uz0UA3SThwOkXWfrX3YoeflZ e/WpIXp5UVJTCLs3vMw4j8RUN4HRr4ELDHB2m1hGLDdSLxsTbaFdUuPi5A+lSo/AByKK6jC ba/wnX6T4pw46CK12PTGpp3ZRCPGmkNl3hanrC7Z7i3JCbTR9NBZ3xtu2hvAKD+sHeG9Qg2 retwUj6AetB8mkjcHyvdQ==
X-UI-Out-Filterresults: notjunk:1;V03:K0:NSzmjq2LYaY=:GXcP1Nmw/+bAvztNYKnEwr Xe7cFcVOFtZUmOvroYBzXmAUf9yBprIPGV96M/qPg+t6OTdVj+6MM6XA+L8aQ12s98k2d+jjE bgjpNco7uPDgvWibDF+R4glhot+Sp8kGa9m75JhA5VP/iBCt5SMJ7yQ9yc75LxieuByUn1Zd/ ebKUciOGp0yqqIfOcIrHnNG+C9rctQpkzk9L2c4LQ4317VaajnFjKoTf0Jb69RiLFPD3F9K8m dnDCsC63EySFM36KGgNFX1c9xEJNiW43tuNrnuJGU/azg/sza7rd3fidHXdgnxyJSchCLE17p VWCReF8QnJBCQIWRMUhwIu1J4zSlgthQAl5QyCeIBsko2Kc5PKHIkQM/6ByWxEku2PDB3bzto 5pWwQUtK/SXlzCtRbT27KYgY322ZUwMKLLTob9aqvwnh/eSAxBEyS6tG0TXjvYQfrknacxA+8 uxPhMeVLDLtRK9Kz9m/dIlUn0z7ItDPBdp2Z4yfnqR5eUvCZpLKZazUAZqmXftP2dNrvmsMbb jr2PU/DD0O0auzHYkYzoVSAyjXWHHcatUDXfGcwtqgWg1jA/KCwu2rn0v6gjYZ0MZdg0CQrd3 WPeufhrhffYJNCaojaibT1c21DKidKp/bQXF83Z0YFZsRYrHnxpm+bKVnBKjH9N/UBEoAjWLN UYZmWgYs5yxZyB5KdMlY5YIDZv4DCY7AUk9doRtKjaTqFevReFibsHxDT+qbtdU3IOisdCvCW g9tLjMUXZ73R8oPdIIECBVNEEVqv6SEOPGVnI52FB6lVgJYxsEq4AOkt4o0aUv64l9OJu5VsN +WCEJxMiqymOti7nuwmPYNigTOy7TANkgdQ2JAFoaQStPgJEc8uzN/pkC0xL2ou6YE6o4JPxG pdWOzsYk4LArEEhrBuIxnfDJbMFuRSeul/aoZ95lwxSqRsmCfq3aOn/ovPKwoHK1t3X7e14hU viR/qoipjx8V9Suo1l690lebnEKfokjlJoSFUe95nDWzQMhcLb38IrSv3uN3DKdVs4jDfh3Ly o1x8ayRqWVaEmix6+bS7v529IeY6uG9Ts45beFp0An1mtajeoQ1oGef4fqIlt/ro72/29dNEl ykbLBqo/5Cy4RNTL9JBBqftn4NoLQtX7xLjOuULBjw73UgS0yJOJnLn/GkySeEWbxhsrPPsxk vkci+fPBpJUwGlabVxEdRgrq68UVa5gETsRRhOVd3pcV6tbXB52vwzCS7clTw4EzE7fC0=
Archived-At: <>
Subject: Re: [tsvwg] path forward on L4S issue #16
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 17 Jun 2020 14:16:19 -0000

Hi Ingemar,

more below in-line.

> On Jun 17, 2020, at 16:08, Ingemar Johansson S <> wrote:
> Hi
> Sebastian 
> Please see inline
> /Ingemar
>> -----Original Message-----
>> From: Sebastian Moeller <>
>> Sent: den 17 juni 2020 15:27
>> To: Ingemar Johansson S
>> <>
>> Cc: Holland, Jake <>om>; Jonathan Morton
>> <>om>; Ingemar Johansson S
>> <>om>;
>> Subject: Re: [tsvwg] path forward on L4S issue #16
>> Hi Ingemar,
>> more below in-line.
>>> On Jun 17, 2020, at 15:20, Ingemar Johansson S
>> <> wrote:
>>> Hi Jake
>>> And thanks for the input.
>>> Given all the input sofar my impression is that It is difficult to say how
>> serious issue 16 really is.
>>> Surely there is a possibility that L4S flows can get an potentially unfair
>> advantage over a congested node that is  RFC3168 capable (and ECN
>> enabled). The question is here when things are deemed as unfair and when
>> starvation is a fact, opinions differ.
>>> Another question is to how large extent there will be legacy RFC3168 ECN
>> capable (and ECN enabled ) bottlenecks that will "never" be updated or
>> replaced. This question is currently difficult to answer.
>>> Yet another aspect is how successful it will be to implement RFC3168
>> bottleneck detection e.g. in TCP Prague, I am quite confident that these will
>> not always be successful.
>> 	[SM] +1
>>> But then, given the current deployment situation and possibility to
>>> update nodes in the networks (OpenWrt routers appear to support
>>> automatic updates)
>> 	[SM] Please accept my apology for not being clear on that topic in the
>> past. Stock OpenWrt does not offer automatic updates, not even notification
>> of update availability at all. There are some vendors, like Evenroute and
>> Turris, that base their own products on top of OpenWrt that do offer
>> automatic updates and as far as I can tell also notifications. But I have no
>> reliable numbers on the prevalence of either stock OpenWrt or update-
>> enabled versions in the field. I also add that a tiny bit of research should allow
>> to confirm this.
> [IJ] Sounds like a contradiction here to me.

	[SM] Why, it is an clarification.

> I was asking for a list of off the shelf routers with OpenWrt support, all these seem to support automatic updates of course limited by some EOL constraints. To me this seems like what the average John or Jane Smith would consider to buy. On top of this we have the few enthusiastic do-it-yourself people who implement OpenWrt in Raspberry PI but that is of lesser interest because a) they are very few and b) they are in the forefront and are thus likely to upgrade pretty fast. 

	[SM] I disagree, we have no numbers to figure out whether assumption a) is true*, and for b) we will first need to demonstrate that any change is actually desirable. L4S is severely lacking in b) so far.

Best Regards

*) I doubt this is correct, as quite a number of OpenWrt users stay on older releases as newer versions will not run well on under-powered devices. So there is going to be a long tail of legacy versions even among OpenWrt users. I base this assessment on supporting users on the OpenWrt forum.

>>> I am not convinced that these need to be 100% bulletproof either.
>> 	[SM] 100% is not achievable, but what we currently have is an effort
>> just designed to get past the objections int the WG instead of actually trying
>> to tackle the issue for real.
>>> /Ingemar
>>>> -----Original Message-----
>>>> From: Holland, Jake <>
>>>> Sent: den 15 juni 2020 20:36
>>>> To: Ingemar Johansson S
>>>> <>rg>; Jonathan
>> Morton
>>>> <>
>>>> Cc: Ingemar Johansson S <>om>;
>>>> Subject: Re: [tsvwg] path forward on L4S issue #16
>>>> Hi Ingemar,
>>>> I posted some summary numbers from some observations I made here:
>> 8660
>>>> 38973a15-6559e7e079a7505b&q=1&e=7a17ee77-c385-4248-836a-
>> 193e64d78d5c&
>> phJ8
>>>> K_CE6is9n7iQy-
>>>> VAZM/
>>>> I didn't say anything about France or Norway, but the numbers I have
>>>> seen were roughly in line with the numbers Jonathan gave.
>>>> The "growing" claim also looks tentatively well-supported, because
>>>> I've seen
>>>> 2 new ASNs since then with a high count of CE-marking paths.
>>>> HTH.
>>>> Best regards,
>>>> Jake
>>>> ´╗┐On 6/10/20, 6:59 AM, "Ingemar Johansson S"
>>>> <> wrote:
>>>> Hi
>>>> Do you have any public sources that confirm the numbers you quote
>>>> below ?. I tried to look up data on this but it surely is not easy.
>>>> Which foras are the vendors that manufacture CPEs active in (if any) ?.
>>>> As regards to endpoints implementing RFC3168, do you refer to servers
>>>> and clients or something else?. My interpretation is servers and
>>>> clients and I don't believe that they are central  to this
>>>> discussion, or do I miss something ?. I understand that traffic
>>>> shaping on outgoing interfaces can be applied in a Linux host but
>>>> don't see why they become a problem especially as there are qdiscs that
>> support dualQ.
>>>> /Ingemar
>>>> PS. I am not yet trying to convince anybody, I am just trying to find
>>>> the answers.
>>>>> -----Original Message-----
>>>>> From: Jonathan Morton <>
>>>>> Sent: den 10 juni 2020 14:58
>>>>> To: Ingemar Johansson S <>
>>>>> Cc: Sebastian Moeller <>de>;
>>>>> Subject: Re: [tsvwg] path forward on L4S issue #16
>>>>>> On 10 Jun, 2020, at 11:35 am, Ingemar Johansson S
>>>>> <> wrote:
>>>>>> 1) How widely are  RFC3168 ECN capable AQMs deployed _and_
>> enabled
>>>> ?
>>>>> Enough for the effects to be visible on about 0.1-1.0% of current
>>>>> Internet traffic - and rising steadily.  That's some combination of
>>>>> endpoints
>>>> negotiating
>>>>> ECN support, and middleboxes implementing an AQM, but not
>>>>> necessarily both on the same path.  Deployment of ECN-enabled AQM is
>>>>> very high across a small number of forward-thinking ISPs, including
>>>>> a very large one
>>>> in
>>>>> France and a major one in Norway.
>>>>> Basically every endpoint out there today implements RFC-3168 ECN,
>>>>> and every Linux endpoint runs an ECN-enabled AQM by default on each
>>>>> Ethernet interface.  If Linux and/or Microsoft had decided to switch
>>>>> it on by
>>>> default a
>>>>> decade ago, rather than hiding it behind a non-default configuration
>>>>> flag,
>>>> we
>>>>> would not be having this conversation, because the deployment
>>>>> numbers would be too big and obvious to ignore.  It's been a long,
>>>>> slow
>>>> chicken-and-
>>>>> egg situation which the major consumer ISPs have essentially refused
>>>>> to
>>>> help
>>>>> resolve.
>>>>> But now there are commercial, off-the-shelf CPE products which
>>>>> anyone can buy and install, and which implement the sort of
>>>>> bufferbloat mitigation measures that most ISPs have not.  I've found
>>>>> at least one of those on the physical retail shelf, here in Finland.
>>>>> I don't have sales numbers, but
>>>> they
>>>>> must account for most of the deployment on ISPs that haven't
>>>>> deployed it themselves.
>>>>>> 2) If they are deployed _and_ enabled, can/will they be updated ?
>>>>> Some can and will.  But some is baked in hardware so would be
>>>>> difficult to update, and some is managed by people/vendors who do
>>>>> not know or care about L4S, so would tend to keep using the old
>>>>> system which is RFC- compliant today.  When was the last time your
>>>>> CPE devices got a firmware update?  So L4S has to be able to cope
>>>>> with the existence of such deployments in at least the medium term.
>>>>> Frankly, if you believe that tunnels are an intractable problem,
>>>>> then this
>>>> is
>>>>> just as bad.  So please finally drop this argument; you're not
>>>>> convincing anyone who matters.
>>>>> - Jonathan Morton