Re: [tsvwg] Gorry Fairhurst Individual thoughts on choosing whether/how to advance ECN work.

[Sebastian Moeller <[e-mail]>]

Dear list, dear Bob,

Tl; dr: In detail debate without much general impact.


> Begin forwarded message:
> 
> From: Bob Briscoe <[e-mail]>
> Subject: Re: [tsvwg] Gorry Fairhurst Individual thoughts on choosing whether/how to advance ECN work.
> Date: May 24, 2020 at 17:54:55 GMT+2
> To: Sebastian Moeller <[e-mail]>
> Cc: [e-mail]
> 
> Sebastian,
> 
> On 23/05/2020 20:02, Sebastian Moeller wrote:
>> Dear Bob,
>> 
>> 
>>> On May 23, 2020, at 17:07, Bob Briscoe <[e-mail]> wrote:
>>> [...]
>>> [BB] I don't think anyone except you is arguing that FQ_CoDel is simpler than the DualQ (because it isn't). You can compare this pseudocode yourself, which also links to the Linux code:
>>> https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-11#appendix-A.1
>> 	I note that the fq-codel RFC contains zero lines of pseudocode, so by this measure it must be infinitely simpler than the dual queue coupled AQM, no?
>> 
>> But joking aside, it seems like an odd choice to compare the complexity of an qdisc in the main-line kernel with an out of line qdisc under active development, but if we attempt to do this, surely 724 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_fq_codel.c (plus 310 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_codel.c) and 746 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_dualpi2.c do not seem to support your hypothesis, unless we want to quibble over individual lines of code here. And yes the runtime cost for fq_codel probably is a bit higher than for dualq, but I would be amazed if we would talk about an order of magnitude (base2) here...
>> 
>> My point is, please refrain from completely unhelpful complexity comparisons by virtue of pseudeocode, especially if not both qdiscs to be compared actually exist in pseudocode in the first place....
> 
> [BB] There are a number of ways one can/should compare complexity. I was not suggesting anyone should count lines of code. I was suggesting Paul should /look/ at the pseudocode, or the code linked from it. You're the one that has turned this into a line counting exercise.

	[SM] What else did you mean when you asked Paul to "compare this pseudocode yourself" after first mentioning both DualQ and fq_codel? I guess I misunderstood what you intended to say then.


> 
> Code complexity (roughly related to lines of code) is important for minimizing bugs, and perhaps for minimizing memory footprint on cost-reduced devices like home gateways. However, for large-scale network equipment, ops per packet are particularly important, and minimizing run-time state is important for most hardware implementations.

	[SM] That is why I wondered, you brought up code complexity in the first place.


> 
>>> The Coupled DualQ solely classifies at the IP layer (ECN field). It consists of two AQM instances:
>> 	[SM] True, until it becomes obvious that without the optional queue protection mechanism dualq severely lacks behind fq_codel in real-world usefulness.
> 
> [BB] I think that statement needs to be tagged with "in the opinion of Sebastian Moeller who has not justified this statement".

	[SM] Well, it goes without saying that all of us can not speak "ex cathedra" here (with exception of the chairs with their official hats on) and hence voice our own opinions, me not less than you. Any specific tagging seems hence redundant, no? That said, I am supporting low-latency interested users as a small part of the SQM effort for several years now, so I am not a really a novice to the whole problem space.

> 
>> Once you add the queue protection to the requirements, your are looking at L4 headers again.
> 
> [BB] Indeed. Finding L4 headers was the most complex part of the whole Low Latency DOCSIS spec, because there are so many possible unusual cases to handle. 

	[SM] Except that at the ISP end-customer edge the only relevant cases are IPv4 and IPv6, but the linux kernel has quite powerful flow dissectors to help you, see e.g. https://elixir.bootlin.com/linux/v4.18.6/source/net/core/flow_dissector.c


> The Linux coupled dualQ implementation handles overload in a much simpler way.

	[SM] Unfortunately, as Koen's recent example demonstrated with rather limited real-world usefulness; in his example three unrelenting fixed rate UDP flows each got ~1/3 of the bottleneck capacity, while all concurrent TCP flows were basically starved. So I would say, the dualQ implementation handles overload in a too simple way, as so often with L4S, "too little, too late". 

> 
> We shall see which turns out to be sufficient. I can understand that many operators will be cautious with a new approach, which is why I agreed to design per-flow QProt for Low Latency DOCSIS, even though I do not believe it will eventually prove necessary. 

	[SM] And again you demonstrate that L4S safety considerations are mostly driven by hope and believe... I would have preferred a design that actually tried to test and robustify the performance under adversarial traffic patterns instead.



> That, to me, is one of the open questions that only an Internet-wide experiment can answer, 

	[SM] This is simply untrue, sure the real internet will generate such adversarial traffic patterns, better than one can do in the lab, but L4S did not even try.
> [...]
> 
>> That said, there is absolutely no reason, why fq_codel could not look only at the IP header for its hashing needs, if you really want that (and interestingly cake offers such flow-definition modes).
>> 
>>> * one instance of an AQM for the Classic queue, with similar complexity to /one/ of the per-flow CoDel instances.
>> 	[SM] +1; it just turned out in the real world that the performance of a single queue AQM simply does not deliver what the end-users actually want from an AQM so that nobody went for a single queue CODEL. I see absolutely no reason, why this should change now with L4S.
> 
> [BB] Because L4S replaces Reno-friendly congestion control, which was the root cause of the problem that made per-flow isolation useful.

	[SM] I object to this classification, that the only or even most value of per-flow-isolation is in isolating the normal TCP-sawtooths from each other. IMHO the value of FQ is that equatable sharing between all active flows is a) simple to reason about, b) avoids starvation and c) might not be the optimal solution for each traffic mix, but is exceedingly unlikely to be a pessimal solution either. Given that most network nodes lack the required information to actually optimize in any meaningful way, I will take alnost never pessimal any old day over L4S's anything goes. If asked why, I just point to Koen's example referenced above.


> 
>> Now, I only talk to end-users, ISPs might see this differently (but ultimately will need to convince their users that L4S is actually an improvement).
>> 
>> 
>>> * one instance of a brain-dead-simple stateless AQM for the L4S queue.
>> 	[SM] Since the two AQMs are coupled, the required state is simply kept on the non-LL side, claiming that it is stateless seems a bit disingenuous, no?
> 
> [BB] The coupling uses a state variable that is already maintained for the non-LL AQM. But adding the LL AQM requires no additional state.

	[SM] That is legerdemain, you need that variable for both legs, so exclusively accounting it to the non-LL side is disingenuous, with equal or more basis in fact you could account it half to each side. But whatever.

> 
>> 
>>> 
>>> I'm seeing that vendors of larger scale equipment /have/ implemented it and operators /have/ plans to adopt it, whereas they didn't adopt FQ_CoDel.
>> 	[SM] Interestingly, end-users interested in low-latency and sharing their access links between, say gaming and streaming, opted for fq_codel and cake. Even though this requires post-bottleneck ingress shaping and is computationally relatively costly. So the relevant market has spoken and adopted FQ_CODEL.
> 
> [BB] You can believe that story if you want. 

	[SM] This is not about what I believe or do not believe, it is that from supporting endusers in debloating their access links with SQM I had some insight in what behavior end-users desire, and currently the best all around solution seems to first share bandwidth equitably between all concurrently active end-user network internal IP addresses and inside of these aggregates share equitably between flows. That is a solution that solves most of the issues users report. Yes, this does not solve 100% of issues for 100% of users, but it is overall a pretty solid and conceptually simple solution, that gives predictability and relative robust and reliable performance, even under bi-directionally saturating loads.

> I think you will find that most end-users who use FQ_CoDel today were not aware they had a choice of anything else. 

	[SM] I have no opinion on the numbers, but the end-users of the SQM package generally gain some insight s in their options. And even on DOCSIS3.1 links often opt for fq_codel/cake on the uplink over the supposed default DOCSIS PIE instance...

> I repeatedly had to pull up people who were relentlessly using the word FQ_CoDel in sentences where the word AQM was appropriate. 

	[SM] These people might exist, I am not one of them so why bring this up?

> I gave up very early on. It continued for years after that.

	[SM] I do not remember you showing up in the relevant OpenWrt user forums at all, which handle did you use?

Best Regards
	Sebastian

> 
> 
> 
> Bob
> 
>>> I'm suggesting an explanation for that (based on what they've told me), not talking theoretic possibilities.
>> 	[SM] Good marketing of the L4S effort? Unless you kick the tires yourself and simply believe the claims in the L4S drafts it seems like the perfect solution, optimal performance for low computational effort. It is only after you realize that this performance is neither robust nor reliable that questions will arise.
>> 
>> 
>>> Another explanation is that L4S enables new interactive applications that are currently infeasible over the WAN.
>> 	[SM To repeat myself, none such applications that are both unfeasible today and are suitable for being run over the best-effort internet have so far been proposed. I am not claiming that L4S might not improve upon the state of the art at all, but this specific claim of currently impossible novel interactive applications seems wishful thinking. Case in point google satia, nvidie gforce now, shadow cloud gaming, all offer interactive real-time reaction-time bound games over the existing internet. That is your market, right there, already working over the existing internet, what are you going to offer?
>> 
>> Best Regards
>> 	Sebastian
>> 
>> 
>>> Regards
>>> 
>>> 
>>> 
>>> Bob
>>> 
>>> -- 
>>> ________________________________________________________________
>>> Bob Briscoe                               http://bobbriscoe.net/
>>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/