Re: [tsvwg] "Pacing" requirement is lost in L4S

[Vasilenko Eduard <[e-mail]>]

Hi Christian,
It is effectively the research of proper ECN for BBR. Very much needed. The more details you would supply to the community - the better.
I have the intention to spend time understanding how different CCAs have proposed to smooth ECN.
Unfortunately in BBR's case, it needs research of code on github - I have not seen a good explanation.
The smoothing algorithm for ECN looks like the last research topic in CCA.

I believe it does not matter ECN(0) or ECN(1) till the point you start to mix BBR with something latency-generating (like CUBIC) in the same queue.
BBR naturally pushes CUBIC out then the necessity for ECN(1) and the separate queue would disappear, ECN(0) would be enough.
Eduard
-----Original Message-----
From: Christian Huitema <[e-mail]> 
Sent: Tuesday, April 23, 2024 18:30
To: Vasilenko Eduard <[e-mail]>; Neal Cardwell <[e-mail]>
Cc: [e-mail]
Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S

To answer Eduard's question, in my experiments, I assume:

- BBR with ECN(1) in the Low latency queue.

It does work pretty well, once the BBR code is updated to measure the average rate of incoming CE marks, and treat that as one of the indicators of congestion.

The update is simple. BBR has a test "IsInflightTooHigh", which checks for packet losses. In the code, I updated that test to also check for the ECN marking rate (the alpha coefficient in Prague), and return "too high" if the marking rate is above a threshold. I also use the test ECN alpha to check whether it is time to exit the startup state, or the probe down state (do not exit if alpha is high).

-- Christian Huitema

On 4/23/2024 7:11 AM, Vasilenko Eduard wrote:
> Hi Neal,
> 
> The logical conclusion from your message is that BBR is better to use with ECN too. No doubt that ECN makes a difference (not a big one – see the next paragraph).
> Let’s compare apple to apple. Compare “Something with ECN” against “BBR with ECN”.
> 
> 
>    *   Ultimately, that queuing occurs because BBR employs substantial filtering of the bandwidth and delay signals it measures. And that's because public Internet paths usually have a wifi, DOCSIS, or cellular hop, and all of those technologies have very noisy delay characteristics.
> Sorry, but filtering/smoothing would be needed for any CCA, even for CCA with ECN (clear congestion signal).
> The problem is that the control loop in the case of CCA may not be faster than the session itself (if the session is not accumulating the big queue – we assume a good CCA algorithm).
> As you said: radio could jump faster (locally) than the smallest control loop latency with ECN.
> But for good control in automation theory, we need the control system to react much faster than the controlled system – It is not possible in CCA.
> 
> Hence, I do believe that mandatory filtering/smoothing would prevent a big difference between:
> 
>    1.  BBR without ECN,
>    2.  BBR with ECN,
>    3.  Something else with ECN.
> Smoothing for a few RTT intervals would be needed in all cases. It would destroy the hypothetical advantage of ECN.
> 
> But the small advantage would be. Hence, better to implement ECN(0). Just I doubt that ECN(1) would be needed (for the separate queue) – especially on a fair apple-to-apple comparison of different CCAs with ECN.
> Unfortunately, the smoothing for a few  RTTs could kill even the desire to have BBR with ECN. The world could stop on BBR without ECN.
> Reminder, a lot of routers out there do not support ECN in AQM. And traffic (logistic curve) is almost at a plateau (+20% left) – fewer and fewer incentives to upgrade.
> 
> PS: By the way, I completely agree with L4S requirement that ECN should be filtered/smoothed on the source, AQM should send ECN as soon as possible. Let’s make it more flexible – software (on the host) would be easy to change.
> Ed/
> From: Neal Cardwell <ncardwell=[e-mail]>
> Sent: Tuesday, April 23, 2024 16:33
> To: Vasilenko Eduard <[e-mail]>
> Cc: [e-mail]
> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
> 
> On Tue, Apr 23, 2024 at 4:13 AM Vasilenko Eduard <vasilenko.eduard=[e-mail]<mailto:[e-mail]>> wrote:
> Hi Neal,
> Big thanks for your answer.
> 
> 1.
> For sure, smart people prefer cooperation to competition. CCAs are competing in reality, would like authors or not.
> However I do not see a value for L4S after BBR finishes the migration (BBR should become the default for Windows, Linux, iOS, and Android to claim that the transition is finished, half of the real traffic is not enough to claim the transition finished).
> Theoretically, L4S has a separate queue where latency is possible to guarantee. It should be better.
> But practically, if the Classic queue would be occupied almost exclusively by BBR, then what would be the additional value of a separate queue?
> I strongly suspect: miserable.
> It is possible to prove only by tests or better pilot deployment.
> If my suspect is true then there would be no motivation for additional hardware requirements of L4S.
> 
> The additional value of L4S is substantial, even when compared to BBR.
> 
> BBR attempts to bound the level of queuing in the bottleneck buffer. However, for multiple BBR flows that bound can cause up to around 1.5*BDP of queuing in the steady-state, with a bottleneck that doesn't support L4S. Ultimately, that queuing occurs because BBR employs substantial filtering of the bandwidth and delay signals it measures. And that's because public Internet paths usually have a wifi, DOCSIS, or cellular hop, and all of those technologies have very noisy delay characteristics.
> 
> L4S provides a precise ECN signal that allows senders to know when there is excessive queuing at the bottleneck. It takes the guesswork out of the equation and allows the queuing caused by flows (multiple flows or a single flow) to be *much* shorter: typically at worst O(milliseconds) of queuing, instead of the 1.5*BDP of queuing you would get from multiple BBR flows in a non-L4S bottleneck.
> 
> best regards,
> neal
> 
> 
> L4S is good when compared to RENO which does not exist anymore.
> L4S should be good compared to CUBIC, but for unknown reasons – it does not try to compare itself to CUBIC.
> 
> 2.
> Section 8 in general is “Security”. People typically do not read that section.
> Requirements are in section 4 - It has nothing about burstiness.
> The quote from section 8.2 has a little sense and may be interpreted very loosely.
> Not many people would be capable of giving the interpretation that you did.
> 
> I did not read the Prague CCA yet because it has the status “personal opinion with zero deployments”.
> It may be that it addresses the burstiness properly.
> I am trying to understand and predict what will happen next, Prague does not look yet as the probable future.
> 
> Eduard
> From: Neal Cardwell 
> <ncardwell=[e-mail]<mailto:[e-mail].
> org>>
> Sent: Friday, April 19, 2024 17:24
> To: Vasilenko Eduard 
> <[e-mail]<mailto:[e-mail]>>
> Cc: [e-mail]<mailto:[e-mail]>
> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
> 
> On Fri, Apr 19, 2024 at 4:39 AM Vasilenko Eduard <vasilenko.eduard=[e-mail]<mailto:[e-mail]>> wrote:
> Hi Experts,
> I see L4S as the "Congestion Control Next Generation from IETF" (that is actually in competition with "Congestion Control Next Generation from Google").
> 
> IMHO BBR is not in competition with L4S.
> 
> BBR is, at its core, about maintaining an explicit model of the network path using whatever signals are available, and using that model to try to achieve low/bounded delay, low loss, and high throughput.
> 
> L4S, IMHO, is largely about creating a low-latency, low-loss, scalable throughput service model (metaphorically a "lane") in Internet bottlenecks, and using ECN to provide a signal to achieve that.
> 
> There is nothing fundamentally at odds about those two models. And once the details of the Prague congestion control algorithm are finalized, one goal (as our team has mentioned for a number of years) is to have a version of BBR that can use L4S signals and coexist with Prague congestion control in the L4S lane of Internet bottlenecks.
> 
> Then I see the important requirement that is missing in L4S.
> 
> The primary requirement for CC is that it "should not grow the buffer on the bottleneck link".
> It is very disputable: is "the Scalable" requirement about it or not? Let's pretend that it is about it.
> 
> Then the next critical requirement is "pacing" which is missed in L4S completely.
> 
> IMHO it is not at all fair or accurate to say that L4S misses the 
> pacing requirement. :-) The pacing requirement is implicit in RFC 9330, at the very least in Section 8.2,  'Latency Friendliness':
>     https://www.rfc-editor.org/rfc/rfc9330.html#section-8.2
> That section says: "Like the Classic service, the L4S service relies on self-restraint to limit the rate in response to congestion. In addition, the L4S service requires self-restraint in terms of limiting latency (burstiness)." The only approach I'm aware of to limit the "rate" and "burstiness" of a flow, and the "latency" that it imposes, is to use pacing.
> 
> And this is explicit in Section 2.5.1, "Packet Pacing", of the Prague congestion control draft, which is part of the L4S suite of documents:
>     
> https://www.ietf.org/archive/id/draft-briscoe-iccrg-prague-congestion-
> control-03.html#section-2.5.1 That section says: "A Prague CCA MUST 
> pace the packets it sends to avoid the queuing delay and under-utilization that would otherwise be caused by bursts of packets..."
> 
> best regards,
> neal
> 
> 
> Kudos to Google because I understood its importance after one local (but big) company tested and investigated BBRv1 (then implemented it).
> After tests, they concluded that pacing is even more important than 
> low latency. (I doubt, probably latency is more important) Imagine that the server would increase the window sharply. The Server may have a 100GE interface. It could generate 10us of traffic as a burst (or even more).
> Intermediate links could be 100GE or even bigger (highly probable), and the burst would travel as it is (without any spreading).
> Then this burst could arrive at 10Mbps subscriber (good performance for shared public WiFi). 0.01ms burst would become 100ms burst.
> It would create many negative consequences for the bottleneck link:
> - tail drop if buffers are not enough
> - guaranteed huge latency
> Hence, we should completely forget about W (window) while discussing CC, we have to use T (time between packets).
> CC next generation "should avoid bursts regulating inter-packet time, not the information permitted in transit".
> 
> Best Regards
> Eduard Vasilenko
> Senior Architect
> Network Algorithm Laboratory
> Tel: [phone]<tel:+7%20985%20910-11-05>