Re: [tsvwg] feedback and thoughts L4S / SCE

Sebastian Moeller <> Sun, 22 November 2020 16:24 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 5477E3A18B2 for <>; Sun, 22 Nov 2020 08:24:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.649
X-Spam-Status: No, score=-1.649 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id Gd9jVLZkCFLW for <>; Sun, 22 Nov 2020 08:24:52 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 177603A18B3 for <>; Sun, 22 Nov 2020 08:24:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=badeba3b8450; t=1606062285; bh=fCTBXBNUCQ3e3v7yx3dSr8709qa2fE+W04Z7Uz7E2mg=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=SaVNNNHIyYzayzwu8hmxRfz+EqL6lBw6oSq4ApQPcJbAGZZjQ6+SFIU7bzCqWLiCa jnz67GubdnS9qKdIvfnB0dNQ4VRDeDvNSJtPOwDRpAy6UCUFKfkXKUmLcFH3CzFI+P AwkCgJTlRGat/N1QVgXbQrk5uybDBZclB1HVoPXk=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [] ([]) by (mrgmx104 []) with ESMTPSA (Nemesis) id 1MWRVb-1kitZ01AYV-00XtK2; Sun, 22 Nov 2020 17:24:45 +0100
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.17\))
From: Sebastian Moeller <>
In-Reply-To: <>
Date: Sun, 22 Nov 2020 17:24:43 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <>
To: Mikael Abrahamsson <>
X-Mailer: Apple Mail (2.3445.104.17)
X-Provags-ID: V03:K1:gfIX0bWoL0eIVlaw+WFStPtvL6K3/T2ACBtLEK89j11xhisonss 6A6T+6sB9mLjCwW+S4davqI0rZ0b2/2SetGRN6wEqGBNjFlpj3NWH/nD8QVINjDmzPPH3cz 47tLWLvbXykVhF/9QBhqj8ga/y6c22oL/mebtEWFZ14ihKYq+QJ5HUKqHF6keZzVn4QlDkA Ukd4Ewht7kPIZZuHJx9oQ==
X-UI-Out-Filterresults: notjunk:1;V03:K0:NyozGBvGUgs=:HWVNIO8pa+USp+dx8vYV0P YTln0XfLL23GOA0v2Ty9YYls4HIURV7TRvYD3FvEQsejz0lMOMONth/lX8f1MWC2iBLN48TCM EBns+s62Zof/VZ+6OoUhY2/LrhYfHmWXFyPdasOSEsJdPcefBiFl8V4EsbQPuhXxU+/LXnp+A ym7Y+j7VI3mspgJh8IWm60n0ux3EkHMu3IA4abvlyBFLTOKA/8+ljKOr64sA2osnG7qaSqvFr JdPC1UqCxu2gqF0DDWpPvvd9IMM4n2DZf3mQvMrfRYhCda7FHdZqtSQqQm1MK+a40hvXxbzG3 c0fgrqxdc8wI0b2KHoGAGEhmgtLzhn7cmI2+MjjonkrirRSue+zHO6qhEctYqZZt3kovoNECM AQg56tvD7jQeI5eb4dMOkcv9a0by+BYzddml1MgcA+hplNOAwI6+DosItOiWnGPFD2I6yswlQ P94SpqjNDT8M+znUjWzwFdi1liHaphxukuqiiYuw/BOcar38lh/cI7AHBnRezTKp/o1F6FQ05 uE7kY7Vkb/juEmaCZgOLbiXd+jcFcUrSVCOMbWmIqogt7tr3akNGavCgBdFEKtuZNnXSFYP5B e5j4n40b2Z0Y6YQZNBj8XiLoHyH1P9y9pfoqMuICTHjlmvTQ6NvJ5/mFzS5+xJimi53yauSKg 0YdcnsnJ+UU8H5S8gwp1iSzVd8GMTA68LBiXUcG8k/CvTRVPJ1t48kuroIZ7CswXAullExZAO 1SeY4oGFrkO4VvPLzvLmt4cGpe7fL/NJ2i4UmLNJn+P2B2n8qXB+5JMzSOVQYeVaVqDutoOv4 xbC9fQqvFnLznJjLDfYRo55ecvDZUWLk+ZDpwIoJfYxtQDec8O3ylgCPJhDtSWcJPcO6UtjeC eetlm8UMcpayX51o1jMg==
Archived-At: <>
Subject: Re: [tsvwg] feedback and thoughts L4S / SCE
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 22 Nov 2020 16:24:54 -0000

Hi Mikhael,

> On Nov 22, 2020, at 15:56, Mikael Abrahamsson <> wrote:
> On Sun, 22 Nov 2020, Sebastian Moeller wrote:
>> just as a data point, OpenWrt switched to fq_codel as default qdisc on all interfaces some years ago.
> And how often is this fq_codel the slowest link in the chain?

	[SM] Same problem as before, we lack reliable numbers.

>> I have no numbers how many OpenWrt devices are in the wild, and how many commercial home router OSes are based on sufficiently modern OpenWrt, but that is some use. The OpenWrt project also has no official counter
> Most of the commercial openwrt based devices will not use CPU based forwarding (required to make fq_codel work) but instead has hardware accelerator (which does not do fq_codel).

	[SM] Well, there is a nss_wfq, and nss_codel, sure that is not the same as nss_fq_codel (see Let's call this a tie, sure, no fq_codel, but wfq (which to my knowledge is wegted fair queueing).

>> of running instances but for example Qualcomm's QCA Software Development Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one would naively assume. That said, the QDSK kernel seems to be quite old (at least Kernel major version 2 seems dead now), and I have no insight which features are back-ported to it.
> If you're using QCA hw-accelerated forwarding then you're not doing fq_codel.

	[SM] Well, I would not and do not not use the accelerated path exactly do to its lack lack of the qdiscs that I would like to use, but I am not representative and will not claim to be so.

>> 	[SM] Again a prediction, not a fact, as I said reasonably recent OpenWrt defaults to fq_codel on all interfaces (that support it) as well as on WiFi which with inceasing access rates more and more often becomes the true bottleneck (no numbers, as that is my prediction, based on my own experience).
> If you're using HW acceleration, then fq_codel isn't used also on wifi.

	[SM] You are sure about that? As far as I know the wifi fq_codel instance lives in the shred 802.11linux infrastructure, is that not used by the QSDK?

>> 	[SM] Given our lack of hard numbers that is telling more about your  judgement than about reality.
> Do you have numbers?

	[SM] Nope, I do not and I admitted to that up front. Doesn't change the fact that you were extrapolating from non-existing numbers here, no?

> How many ISP provided routers out there actually use fq_* ? ISP provided router is the most common in use.

	[SM] Again, no numbers I can offer.

>> 	[SM] And that is pretty harsh. Also unhelpful teminology, "irrelevant" I would accept, but invalid is simply the wrong quality to argue about.
> Fine.
>> 	[SM] And here we are again at our lack of numbers... As before I share your intuition, but it is not a fact.
>> "Fun" fact TCP Prague really is not at all suited to a FIFO world at all, as Pete demonstrated.
> Do you have better numbers?

	[SM] Sorry no, and as I said I share your intuition, but that tells more about my intuition than about reality ;)

>>> I also doubt we'll see widely deployed FQ going forward because of the cost of doing this in hw (and most devices are hw accelerated).
>> 	[SM] An often heard claim that suffers from the fact that that has simply never been tried at scale, so the cost argument reflects the status quo not necessarily actual economic "laws", no?
> Do you know of a FQ_* implementation in hardware?

	[SM] No, looking at the literature, there are implementations for FPGAs though, but I am not sure how close to reality these are. But as I said, due to lack of demand (and lack of paying extra for a new feature) it is not clear to me whether the lack of FQ in commercial routers is driven by the task being technically infeasible or by it being economically unattractive.

>> 	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).
> Thanks for ack:ing that.

	[SM] In case this is in doubt, I am trying to assess all solutions to low latency networking by the same yards stick and try to be consistent with my requirements (if not I try to explain a position that looks inconsistent at first).
	And, to repeat myself, and without hard numbers to back this up, I think your offered frequency estimates are quite reasonable.

>> QUESTION: Where do you think most choke points live, at the ISP-end-user boundary, deeper in the ISP's network, or even at exchange points? My own limited experience points to the ISP-end-user edge, but that is purely anecdotal.
> The choke point is most often the customer last-mile access line or the wifi hop.

	[SM] Good. That clearly indicates, that whether or not big-iron back-bone routers ever develop AQMs or not is not going to be the critical point for HFCC's feasibility! And then I would say SQM in OpenWrt demonstrates that FQ and compeent AQM at the ISP to end-user transition is feasible up to Gbps rates with gear below the $100 mark (a raspberry pi4B has enough CPU cycles for traffic shaping just below line rate, and it is just an USB3 ethernet dongle short of being a wired router)*. 
	Sure that currently is a) not a solution for most "normal" end-users, and b) is probably too expensive compared to what ISPs pay for the CPE they distribute (especially since it lacks adequate WiFi). But it does show that at that location FQ is not a pie-in-the-sky impossibility, nor prohibitively expensive.
	That it is IMHO possible and viable, does not change that I agree with you, that currently is is a small minority of links that are operated in that way. Introduction of HFCC would be a great possibility to change that ;)

Best Regards

*) I do not claim to be fully objective here, I am involved in the SQM effort.

> -- 
> Mikael Abrahamsson    email: