Re: [tsvwg] feedback and thoughts L4S / SCE

[Sebastian Moeller <[e-mail]>]

Hi Mikael,

just as a data point, OpenWrt switched to fq_codel as default qdisc on all interfaces some years ago. I have no numbers how many OpenWrt devices are in the wild, and how many commercial home router OSes are based on sufficiently modern OpenWrt, but that is some use. The OpenWrt project also has no official counter of running instances but for example Qualcomm's QCA Software Development Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one would naively assume. That said, the QDSK kernel seems to be quite old (at least Kernel major version 2 seems dead now), and I have no insight which features are back-ported to it.



> On Nov 20, 2020, at 14:33, Mikael Abrahamsson <swmike=[e-mail]> wrote:
> 
> 
> Hi,
> 
> I'm trying to catch up on the work has been done, but I thought I'd give some thoughts / feedback on what I see:
> 
> https://github.com/heistp/l4s-tests
> 
> "L4S transports experience intra-flow latency spikes at RFC3168 bottlenecks, particularly with the widely deployed fq_codel"
> 
> FQ_CODEL isn't widely deployed.

	[SM] Given your lack of numbers about OpenWrt that is a pretty precise statement that is hard to prove or disprove. But for a change I actually agree that your intuition here, but I want to point out that it is a (likely) prediction and not a fact. It is important in this discussion to separate predictions from fact though, so let's be open about what we know and what we do not know.


> Yes, it's enabled in the linux kernel but very seldom at the bw chokepoint of the connection.

	[SM] Again a prediction, not a fact, as I said reasonably recent OpenWrt defaults to fq_codel on all interfaces (that support it) as well as on WiFi which with inceasing access rates more and more often becomes the true bottleneck (no numbers, as that is my prediction, based on my own experience).


> I do not agree with the above statement at all.

	[SM] Given our lack of hard numbers that is telling more about your  judgement than about reality.

> Thus any benchmarking vs FQ_ anything is invalid.

	[SM] And that is pretty harsh. Also unhelpful teminology, "irrelevant" I would accept, but invalid is simply the wrong quality to argue about. 


> What's mostly used today are typically simple FIFOs, sometimes diffserv enabled with multiple FIFOs.

	[SM] And here we are again at our lack of numbers... As before I share your intuition, but it is not a fact. 
"Fun" fact TCP Prague really is not at all suited to a FIFO world at all, as Pete demonstrated.

> 
> This means the tunnel argument against any FQ or non-FQ AQM is invalid.

	[SM] Again validity seems the wrong yardstick, but I take your point.


> I also doubt we'll see widely deployed FQ going forward because of the cost of doing this in hw (and most devices are hw accelerated).

	[SM] An often heard claim that suffers from the fact that that has simply never been tried at scale, so the cost argument reflects the status quo not necessarily actual economic "laws", no?


> 
> "Deployments of fq_codel
> The fq_codel qdisc has been in the Linux kernel since version 3.6 (late 2012) and is now in widespread use in commercial routers (e.g. Ubiquiti EdgeMAX and UniFi products)"
> 
> This is misleading. Yes, EdgeMAX supports FQ_CODEL if you turn off hw_acceleration, bringing down the performance to 10% or less of what the hw acceleration yields. I do not agree that it's in "widespread" use.

	[SM] Side-note, the same hardware acceleration also disallows quite a large number of features of the Linux kernel's networking features, and IMHO is a step in the wrong direction, allowing chep-ish devices to shine in stereotyped tests, but fail in real world use once the user/owner wants to step beyond the middle-of-the-road feature set supported by the accelerator. THere is a reason why hardware acceleration features have a hard time being accepted in the Linux kernel...
	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).


> 
> I'd gladly take data to prove me wrong, that FQ_CODEL is in widespread use at Internet bw chokepoints.

	[SM] As I am sure that Pete will take data disproving the opposite claim you make, but that leaves us none the wiser.
QUESTION: Where do you think most choke points live, at the ISP-end-user boundary, deeper in the ISP's network, or even at exchange points? My own limited experience points to the ISP-end-user edge, but that is purely anecdotal.


> I'd be surprised if it was enabled on more than 1% of residential connections.

	[SM] This is a good question. Anybody here can offer an sufficiently robust estimate of the total number of residential connections? 


> 
> On to L4S.
> 
> Now, taking above statement and looking at what L4S is doing, for 1-2 decades going forward L4S enabled transport will often encounter FIFOs, mostly without ECN capability at all. They need to be proven safe in this operational reality. From what I can see, this is still not the case?

	[SM] Why only bother about safety here, why not also expect actual delivering on its promises? ATM the data as presented by Pete indcates that L4S as currently designed and implemented will give robustly and reliably rate and latency advantages of L4S versus non-L4S traffic (while claiming only to give a latency edge), and will robustly and reliably give an advantage of short RTT L4S flows over longer RTT L4S (in significant excess over the known and expected RTT bias as seen in FIFOs and still significanly stronger than in other single queue AQMs like the non-L4S PI instance). in short all L4S currently does is establish a fast lane for short RTT flows (which accidentally is al that team L4S ever tested, so I assume that this is not accidental but a non verbalized design goal).


> 
> Going forward:
> 
> In order to progress to a solution, we need to have participants agree on what the world looks like right now, what's doable in the next few years, and what's required to handle existing/near term reality. SCE proponents have to stop arguing that we'll have FQ_ anything near term, or that this is in wide use. It's not. L4S proponents need to prove that their proposals are safe in a world of FIFOs,

	[SM] Addendum, we should also ecpect that L4S transports actually work reasonably well for actually transporting data in the existing mostly FIFO internet. L4S comes with known side-effects and wants a special treat (ECT(1)) so should actually be expected to deliver on it promises.


> and accept that this is going to be a reality for years to come. AQMs aren't deployed in a hurry, we're talking many years to deploy on the wider Internet.
> 
> We're spending the last bit in the IP header and we'd better get it right. In order to do that, I'd like to see realism and pragmatism in the reasoning.

	[SM] +1. A discussion that is closer tied to the existing data and that does not take "claims" in the drafts as facts would also help to make the discussion more relevant.

Regards
	Sebastian


> 
> -- 
> Mikael Abrahamsson    email: [e-mail]
>