Re: [tsvwg] feedback and thoughts L4S / SCE

Sebastian Moeller <> Sun, 22 November 2020 14:46 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 6946F3A177D; Sun, 22 Nov 2020 06:46:11 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.649
X-Spam-Status: No, score=-1.649 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id hYWG2vaMuFRN; Sun, 22 Nov 2020 06:46:07 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 05B4E3A177A; Sun, 22 Nov 2020 06:46:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1606056363; bh=wKoYMQY5h555GNJ2ULSiJ+miDZA1SgWc1anL6jaH3og=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=hd3RY/RLzvzCGALqBK0N5kA+64JjErjmroxQ7DGOZNT1M4/b9yfFAUIYA9Oe+0dDN wO2SREhCCknRp8M3liqcIo72o8w1NaIbuHkUuUq6HiX6xclJDqI6c+3B0Gt01Wdwgn LYT8wZ+jzJ0Yj02A3RXJ7WMOsuxV5bjtZoZwvhJk=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (mrgmx105 []) with ESMTPSA (Nemesis) id 1M4JmT-1kh7bm2eYr-000InO; Sun, 22 Nov 2020 15:46:03 +0100
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.17\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Sun, 22 Nov 2020 15:46:02 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <>
To: Mikael Abrahamsson <>
X-Mailer: Apple Mail (2.3445.104.17)
X-Provags-ID: V03:K1:d7p506bLLoesRFzCQwBAAL9uQUhQ3wIQXHhRJmrOYzbDTCoFmbr 6JPoy0OxLfOTX8saHf9//Cg+q3PShP+KngO5TeStvDQQ38VQcJhQwpl9hX6ROOYmONOK68a Pz2Zg/UI7y/rL4jfMquk19dzn7z8IX1/YCMXulCMpQiwkIXOc4H9ry+13KcHD3v4q8c1IzF B5MbJPgEp/AoXMPdUdO9g==
X-UI-Out-Filterresults: notjunk:1;V03:K0:8ZVCDaNJH58=:SQ92Hb0EH+xqEA/7BQ6iSb YMSwVqzcXqtTxY7zeIYmHno8rezgN3FDIe6H7gajHtfLrX8Y9A4D6Y3aRHXfn6s2Z/iWOrht6 D6P2vyCQLyr/4wpqrCZt80l6Nv3fzu56EtvKbNst2xPSZii4in7kpmYjGxqXgIr/F0j425UlJ tSIk+V/4RY7li2CX3CrRIbELscmREbv+nmaTxpKOg5VIB0z7Rrqpv+qxCoR0Vg4jo5h5lwZR9 mNRnTnFooAFhEKGhva7GyjCuPSyj/C6D98e2ipZm0aQUPq1PXfq5JAHo35UQEn5yoe0pjuGO7 Dtf6UOdLOXfc5PaLO9lNMwuqM3IrcWi8CukptJ+TUmbiXHVo4++L2JLE8EHR1IU8HP+TZUPI4 3Xchxy0UM4TeRue+a0eE8q76EcpsGnburLPeBhxB0PrQkfa6u2Sw6wskYoddiVg+tj5/TAcPi YwP+oxJyG7QLIjzBaEnhxR2cWBgFArtkpmGmeozmu+A66tf2EhBgETxBitYchtQcpxvbiJD9M UV2oZ0oBkIqTPNvjVBxcrACNLOKYZEEzZsuF+r9QXG8F+chNxVB3vjtsc1NSZnncnsoMWe6bj Hpoe7jDBFGvJp2zO+WGx/zUtHbS2l9nLYfuIdQ5++SxPBI3aXBPeOTbgZ1wK04hO1PT7pYCyF f4hKk4m8y/3DkQRGlJvS0JlHY85vymn2TYmAdJJ8jEKMkbGRlhdkkp0govSXXzqDxjR0kKTBT 73xt6hCel4boT0+k3dsge27IOMGtdMp0w9TDjA729PM33ckktNIB2m1G7urfn0jC36dlj36lQ E95xIZT/3RFEtvCeEr/RuVj1QCJ7m/ImjUd0R8RU3UBIR+KCFeSMhUzu/AAJp0DYkwzofbPie LZr8vMoLyO2D+yQuxtWQ==
Archived-At: <>
Subject: Re: [tsvwg] feedback and thoughts L4S / SCE
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 22 Nov 2020 14:46:12 -0000

Hi Mikael,

just as a data point, OpenWrt switched to fq_codel as default qdisc on all interfaces some years ago. I have no numbers how many OpenWrt devices are in the wild, and how many commercial home router OSes are based on sufficiently modern OpenWrt, but that is some use. The OpenWrt project also has no official counter of running instances but for example Qualcomm's QCA Software Development Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one would naively assume. That said, the QDSK kernel seems to be quite old (at least Kernel major version 2 seems dead now), and I have no insight which features are back-ported to it.

> On Nov 20, 2020, at 14:33, Mikael Abrahamsson <> wrote:
> Hi,
> I'm trying to catch up on the work has been done, but I thought I'd give some thoughts / feedback on what I see:
> "L4S transports experience intra-flow latency spikes at RFC3168 bottlenecks, particularly with the widely deployed fq_codel"
> FQ_CODEL isn't widely deployed.

	[SM] Given your lack of numbers about OpenWrt that is a pretty precise statement that is hard to prove or disprove. But for a change I actually agree that your intuition here, but I want to point out that it is a (likely) prediction and not a fact. It is important in this discussion to separate predictions from fact though, so let's be open about what we know and what we do not know.

> Yes, it's enabled in the linux kernel but very seldom at the bw chokepoint of the connection.

	[SM] Again a prediction, not a fact, as I said reasonably recent OpenWrt defaults to fq_codel on all interfaces (that support it) as well as on WiFi which with inceasing access rates more and more often becomes the true bottleneck (no numbers, as that is my prediction, based on my own experience).

> I do not agree with the above statement at all.

	[SM] Given our lack of hard numbers that is telling more about your  judgement than about reality.

> Thus any benchmarking vs FQ_ anything is invalid.

	[SM] And that is pretty harsh. Also unhelpful teminology, "irrelevant" I would accept, but invalid is simply the wrong quality to argue about. 

> What's mostly used today are typically simple FIFOs, sometimes diffserv enabled with multiple FIFOs.

	[SM] And here we are again at our lack of numbers... As before I share your intuition, but it is not a fact. 
"Fun" fact TCP Prague really is not at all suited to a FIFO world at all, as Pete demonstrated.

> This means the tunnel argument against any FQ or non-FQ AQM is invalid.

	[SM] Again validity seems the wrong yardstick, but I take your point.

> I also doubt we'll see widely deployed FQ going forward because of the cost of doing this in hw (and most devices are hw accelerated).

	[SM] An often heard claim that suffers from the fact that that has simply never been tried at scale, so the cost argument reflects the status quo not necessarily actual economic "laws", no?

> "Deployments of fq_codel
> The fq_codel qdisc has been in the Linux kernel since version 3.6 (late 2012) and is now in widespread use in commercial routers (e.g. Ubiquiti EdgeMAX and UniFi products)"
> This is misleading. Yes, EdgeMAX supports FQ_CODEL if you turn off hw_acceleration, bringing down the performance to 10% or less of what the hw acceleration yields. I do not agree that it's in "widespread" use.

	[SM] Side-note, the same hardware acceleration also disallows quite a large number of features of the Linux kernel's networking features, and IMHO is a step in the wrong direction, allowing chep-ish devices to shine in stereotyped tests, but fail in real world use once the user/owner wants to step beyond the middle-of-the-road feature set supported by the accelerator. THere is a reason why hardware acceleration features have a hard time being accepted in the Linux kernel...
	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).

> I'd gladly take data to prove me wrong, that FQ_CODEL is in widespread use at Internet bw chokepoints.

	[SM] As I am sure that Pete will take data disproving the opposite claim you make, but that leaves us none the wiser.
QUESTION: Where do you think most choke points live, at the ISP-end-user boundary, deeper in the ISP's network, or even at exchange points? My own limited experience points to the ISP-end-user edge, but that is purely anecdotal.

> I'd be surprised if it was enabled on more than 1% of residential connections.

	[SM] This is a good question. Anybody here can offer an sufficiently robust estimate of the total number of residential connections? 

> On to L4S.
> Now, taking above statement and looking at what L4S is doing, for 1-2 decades going forward L4S enabled transport will often encounter FIFOs, mostly without ECN capability at all. They need to be proven safe in this operational reality. From what I can see, this is still not the case?

	[SM] Why only bother about safety here, why not also expect actual delivering on its promises? ATM the data as presented by Pete indcates that L4S as currently designed and implemented will give robustly and reliably rate and latency advantages of L4S versus non-L4S traffic (while claiming only to give a latency edge), and will robustly and reliably give an advantage of short RTT L4S flows over longer RTT L4S (in significant excess over the known and expected RTT bias as seen in FIFOs and still significanly stronger than in other single queue AQMs like the non-L4S PI instance). in short all L4S currently does is establish a fast lane for short RTT flows (which accidentally is al that team L4S ever tested, so I assume that this is not accidental but a non verbalized design goal).

> Going forward:
> In order to progress to a solution, we need to have participants agree on what the world looks like right now, what's doable in the next few years, and what's required to handle existing/near term reality. SCE proponents have to stop arguing that we'll have FQ_ anything near term, or that this is in wide use. It's not. L4S proponents need to prove that their proposals are safe in a world of FIFOs,

	[SM] Addendum, we should also ecpect that L4S transports actually work reasonably well for actually transporting data in the existing mostly FIFO internet. L4S comes with known side-effects and wants a special treat (ECT(1)) so should actually be expected to deliver on it promises.

> and accept that this is going to be a reality for years to come. AQMs aren't deployed in a hurry, we're talking many years to deploy on the wider Internet.
> We're spending the last bit in the IP header and we'd better get it right. In order to do that, I'd like to see realism and pragmatism in the reasoning.

	[SM] +1. A discussion that is closer tied to the existing data and that does not take "claims" in the drafts as facts would also help to make the discussion more relevant.


> -- 
> Mikael Abrahamsson    email: