Re: [tsvwg] feedback and thoughts L4S / SCE

[Sebastian Moeller <[e-mail]>]

Hi Mikhael,


> On Nov 22, 2020, at 15:56, Mikael Abrahamsson <[e-mail]> wrote:
> 
> On Sun, 22 Nov 2020, Sebastian Moeller wrote:
> 
>> just as a data point, OpenWrt switched to fq_codel as default qdisc on all interfaces some years ago.
> 
> And how often is this fq_codel the slowest link in the chain?

	[SM] Same problem as before, we lack reliable numbers.


> 
>> I have no numbers how many OpenWrt devices are in the wild, and how many commercial home router OSes are based on sufficiently modern OpenWrt, but that is some use. The OpenWrt project also has no official counter
> 
> Most of the commercial openwrt based devices will not use CPU based forwarding (required to make fq_codel work) but instead has hardware accelerator (which does not do fq_codel).

	[SM] Well, there is a nss_wfq, and nss_codel, sure that is not the same as nss_fq_codel (see https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/drivers/net/ethernet/atheros/nss-qdisc). Let's call this a tie, sure, no fq_codel, but wfq (which to my knowledge is wegted fair queueing).


> 
>> of running instances but for example Qualcomm's QCA Software Development Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one would naively assume. That said, the QDSK kernel seems to be quite old (at least Kernel major version 2 seems dead now), and I have no insight which features are back-ported to it.
> 
> If you're using QCA hw-accelerated forwarding then you're not doing fq_codel.

	[SM] Well, I would not and do not not use the accelerated path exactly do to its lack lack of the qdiscs that I would like to use, but I am not representative and will not claim to be so.

> 
>> 	[SM] Again a prediction, not a fact, as I said reasonably recent OpenWrt defaults to fq_codel on all interfaces (that support it) as well as on WiFi which with inceasing access rates more and more often becomes the true bottleneck (no numbers, as that is my prediction, based on my own experience).
> 
> If you're using HW acceleration, then fq_codel isn't used also on wifi.

	[SM] You are sure about that? As far as I know the wifi fq_codel instance lives in the shred 802.11linux infrastructure, is that not used by the QSDK?


> 
>> 	[SM] Given our lack of hard numbers that is telling more about your  judgement than about reality.
> 
> Do you have numbers?

	[SM] Nope, I do not and I admitted to that up front. Doesn't change the fact that you were extrapolating from non-existing numbers here, no?


> How many ISP provided routers out there actually use fq_* ? ISP provided router is the most common in use.

	[SM] Again, no numbers I can offer.


> 
>> 	[SM] And that is pretty harsh. Also unhelpful teminology, "irrelevant" I would accept, but invalid is simply the wrong quality to argue about.
> 
> Fine.
> 
>> 	[SM] And here we are again at our lack of numbers... As before I share your intuition, but it is not a fact.
>> "Fun" fact TCP Prague really is not at all suited to a FIFO world at all, as Pete demonstrated.
> 
> Do you have better numbers?

	[SM] Sorry no, and as I said I share your intuition, but that tells more about my intuition than about reality ;)


> 
>>> I also doubt we'll see widely deployed FQ going forward because of the cost of doing this in hw (and most devices are hw accelerated).
>> 
>> 	[SM] An often heard claim that suffers from the fact that that has simply never been tried at scale, so the cost argument reflects the status quo not necessarily actual economic "laws", no?
> 
> Do you know of a FQ_* implementation in hardware?

	[SM] No, looking at the literature, there are implementations for FPGAs though, but I am not sure how close to reality these are. But as I said, due to lack of demand (and lack of paying extra for a new feature) it is not clear to me whether the lack of FQ in commercial routers is driven by the task being technically infeasible or by it being economically unattractive.


> 
>> 	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).
> 
> Thanks for ack:ing that.

	[SM] In case this is in doubt, I am trying to assess all solutions to low latency networking by the same yards stick and try to be consistent with my requirements (if not I try to explain a position that looks inconsistent at first).
	And, to repeat myself, and without hard numbers to back this up, I think your offered frequency estimates are quite reasonable.



> 
>> QUESTION: Where do you think most choke points live, at the ISP-end-user boundary, deeper in the ISP's network, or even at exchange points? My own limited experience points to the ISP-end-user edge, but that is purely anecdotal.
> 
> The choke point is most often the customer last-mile access line or the wifi hop.

	[SM] Good. That clearly indicates, that whether or not big-iron back-bone routers ever develop AQMs or not is not going to be the critical point for HFCC's feasibility! And then I would say SQM in OpenWrt demonstrates that FQ and compeent AQM at the ISP to end-user transition is feasible up to Gbps rates with gear below the $100 mark (a raspberry pi4B has enough CPU cycles for traffic shaping just below line rate, and it is just an USB3 ethernet dongle short of being a wired router)*. 
	Sure that currently is a) not a solution for most "normal" end-users, and b) is probably too expensive compared to what ISPs pay for the CPE they distribute (especially since it lacks adequate WiFi). But it does show that at that location FQ is not a pie-in-the-sky impossibility, nor prohibitively expensive.
	That it is IMHO possible and viable, does not change that I agree with you, that currently is is a small minority of links that are operated in that way. Introduction of HFCC would be a great possibility to change that ;)

Best Regards
	Sebastian


*) I do not claim to be fully objective here, I am involved in the SQM effort.


> 
> -- 
> Mikael Abrahamsson    email: [e-mail]