[tsvwg] fq_codel deployment size

[Dave Taht <[e-mail]>]

"[e-mail]" <[e-mail]> writes:

> On Thursday, November 7, 2019, 3:35:06 AM GMT, Jonathan Morton
> <[e-mail]> wrote: 
>
>> > - if the CoDel queue is upgraded to perform Immediate AQM on L4S
> flows, the latency spike can be largely avoided.
>
>> This is not relevant, since the point of this exercise is to
> establish the extent of L4S' compatibility with existing, unmodified
> networks, in which Codel happens to be the most widely deployed > AQM.
> You cannot reasonably expect the entire Internet to "upgrade" to L4S
> compatible AQMs before you can safely begin deployment.
>
> I wonder if Codel actually is the most widely deployed AQM. This is

Codel is almost never deployed by itself.

FQ-codel is the most widely deployed, enabled AQM and so far as
I can tell, the only one with ECN enabled by default.

It is the default in apple's IOS (you have to jailbreak your iphone to
see it), on all interfaces, the default in OSX, the default in nearly
every linux distribution. (the last to add it was RHEL8). One thing
I've lost a lot of sleep over is the heavy use of containers and
network namespaces nowadays (billions of instances), where I fear
fq_codel is the only thing keeping the internet from exploding as
the box is essentially a "router" when used as such. 

The wifi version is the default (in linux) on QCA's ath9k and ath10k
chips, the mt76, and most recently, many of intel's chips as well.

So, that's kind of over a billion boxes.

Merely counting up the major wifi vendors shipping a version - google
chromebooks, google wifi, google fiber, evenroute, eero, meter, meraki,
tp-link, netgear, etc...

I hate to wave figures around without doing detailed research, but I can
comfortably say the wifi version is in the 10s of millions a year....

As for using fq_codel in QoS/SQM systems, it's got huge penetration, but
I doubt more than a fraction of a percent of users (outside the gaming
and clued business markets using ubnt and/or pfsense) enable it.

Free.fr shared details with me of a percentage of their deployment in a
few million dsl home routers. Of the 36,000 samples that I have, the top
level bits were:

* It's on the uplink only as the downlink invisibly multiplexes tv
* It is configured in the 3 class QoS system so common in SQM,
  with fq_codel on each class. Prioirity, best effort and background
* They have a *tight* - bql-like - interface to their DSL chip, so
  they don't need to shape at all, system automagically has pushback,
  copes with dsl problems, thus extremely low cpu costs.
  (I have extreme frustration with not seeing bql deploy widely elsewhere)
* 48% had exherted a CE at some point in this boot cycle. 
* A fairly typical drop and mark ratio is about 20/1 - meaning the
  majority of connections aren't using ecn.
* The customers are happy. Since 2012 :)
* They wish strongly that they could somehow get this into their fiber
* deployment, instead they use small fifos and hope for the best.

Me randomly picking a piece of data from their reference QoS
implemntation looks like this. You can infer the speed of the uplink
by the size of the target variable. (target is set to 1.5MTU at their
link rate)

qdisc prio 1: root refcnt 2 bands 2 priomap  1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1
 Sent 25418296010 bytes 152822679 pkt (dropped 329184, overlimits 0 requeues 481924) 
 backlog 0b 0p requeues 481924 
qdisc fq_codel 8001: parent 21:1 limit 10240p flows 1024 quantum 1538 target 61.0ms interval 100.0ms ecn 
 Sent 25308497918 bytes 152490355 pkt (dropped 327079, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  maxpacket 44385 drop_overlimit 0 new_flow_count 17308870 ecn_mark 20738
  new_flows_len 0 old_flows_len 9
qdisc drr 21: parent 1:2 
 Sent 25418296010 bytes 152822679 pkt (dropped 329184, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc fq_codel 8002: parent 21:2 limit 10240p flows 1024 quantum 1538 target 61.0ms interval 100.0ms ecn 
 Sent 109798092 bytes 332324 pkt (dropped 2105, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  maxpacket 1500 drop_overlimit 0 new_flow_count 369 ecn_mark 0
  new_flows_len 1 old_flows_len 4
qdisc fq_codel 11: parent 1:1 limit 10240p flows 1024 quantum 1538 target 61.0ms interval 100.0ms ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

The router itself runs multiple services locally and its tcp is cubic
with ecn enabled.

I know of about 14 other ISPs that have done some deployment but they
aren't willing to talk about it. One was overjoyed to share data with me
(as most of their network was fully fq_codeled and even "caked" at the
peering points) until they ran some rrul tests and realized how bloated
(seconds) some of their wireless services were. I figure they'll get
back to me after they fix that....

I've been looking for someone to collaborate with on a paper on this
dataset for some time. (I've got another fun research project going on
which is looking at bufferbloat in coffeeshops. Aside from google
starbucks, it's pretty dismal worldwide)

> not to dispute your point, which is correct - proposing an update to
> Codel doesn't addess existing deployments..
> It's just that I'm aware that there is an awful lot of kit out there
> which has an implementation of RED (or WRED) in hardware - eg for
> TR-059 [1], which mandates WRED support in BRAS. That's from having
> worked at a silicon vendor, so it doesn't give me visibility of actual
> usage. I understand a lot of it isn't switched on, but if even a small

I wish it was switched on, but it seems too hard to tune for most.

> fraction of it was, it would be a pretty big deployment. It would be
> interesting if anyone has actual numbers. . Of course, to be relevant

I have (in the US) only seen one (dsl) ISP that appears to have enabled
WRED in the BRAS, but my data set is limited,

> to this discussion, it would need to be configured to do ECN marking
> as well. 


> [1] https://www.broadband-forum.org/technical/download/TR-059.pdf
>
> Alex