Re: [tsvwg] "Pacing" requirement is lost in L4S

["Bless, Roland (TM)" <[e-mail]>]

Hi Eduard,

[sorry messed up your first name], see inline.

On 19.04.24 at 15:16 Vasilenko Eduard wrote:
> Thanks. Looks like I have understood now.
> Pacing or rate-limiting is about what is the averaging interval.
> 
> The example that I have presented initially (bump to the low-speed link) needs pacing (on the small interval).

> I still do not understand why it is so mandatory to have the "window" concept.

It is not mandatory, but useful IMHO, because
window-based congestion control
  - automatically achieves stability even if window size is not perfect
  - implicitly adjusts the sending rate
  - limits the amount of inflight data -> and thus the queue length
  - if the window is too large: you get a standing queue at the 
bottleneck, but no instability (i.e., steadily growing queue)

> Because if CC is capable of limiting the bottleneck queue in any way then "unlimited queue growth" is not possible.

That is correct, but the question is _how_ the bottleneck queue can be
limited. Simply picking a "suitable" sending rate is not sufficient
since burstiness may cause unlimited queue growth. As I said:
at a 100Mbit/s bottleneck, two senders with an average data rate of
_exactly_ 50Mbit/s but exponentially distributed sending behaviors
show enough burstiness to cause unlimited queue growth (this is also 
confirmed by queuing theory).

> I believe that the clear ECN signal is very good for this purpose, but observing the RTT looks not bad too (as BBR has proved).

Yes, using multiple congestion signal are always good. BBR does not use
the RTT as congestion signal BTW, which seems to be a common
misunderstanding. BBR uses an estimation of RTT_min to calculate the
BDP for a flow. BBRv2/3 use ECN and packet loss (usually with threshold 
2%) as congestion signals, but the designers deliberately do not use the
measured RTT as it may include too much noise.

> If one has any of these then one does not need a "window".
> 
> "Congestion window" is a very good mechanism to get a bigger latency.

Nope, generally speaking, if you use the perfectly fitting window size
there will be no queuing delay (if combined with pacing to avoid
micro-bursts).
We have designed TCP-LoLa [*] that is delay- and window-based
and able to limit the queuing delay to a configured bound
(e.g., 5ms for all the flows at the same bottleneck).
[*] https://ieeexplore.ieee.org/document/8109356

Regards,
  Roland

> Anyway, the fact that L4S is not talking about how to mitigate burstiness is a big hole in the "Next Generation Congestion Control".
> Eduard
> -----Original Message-----
> From: Bless, Roland (TM) <[e-mail]>
> Sent: Friday, April 19, 2024 13:57
> To: Vasilenko Eduard <[e-mail]>; [e-mail]
> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
> 
> Hi Vasilenko,
> 
> On 19.04.24 at 12:01 Vasilenko Eduard wrote:
>> 1.
>> CC may measure RTT, understand buffer growth, and increase the interval between packets.
> 
> Delay-based CC is a different aspect and orthogonal to pacing, but typically pacing is also beneficial for them.
> 
>> Reaction by W and T looks the same effective against loaded buffer.
>> Actially W is easy to re-calculate to T. I do not see a problem.
> 
> As I said, the difference is that a sheer existence of a congestion window has a self-stabilizing effect on buffer occupation, whereas two senders that simply send with perfect average rate shares towards a bottleneck (but slightly bursty sending rate, e.g., packet sending times from an exponential distribution) may cause unlimited queue growth.
> 
>> 2.
>> It looks like I use the wrong terminology because "pacing" for me is exactly the situation when information is sent on some timer (that may change slowly).
>> But you use "pacing" in combination with "BBR rate-based sending" which looks to me like "pacing for pacing".
>> It looks like "pacing" is something different for you.
>> Probably it is my fault. But I have not understood.
> 
> Personally, I actually like to distinguish between rate-based sending and pacing:
> BBR calculates a target sending rate and uses paced sending to avoid burstiness even on smaller time scales. You could effectively use the same target sending rate over a larger interval, e.g., RTT_min and send it in a much more bursty manner.
> That would still be rate-based, but without pacing.
> 
> Regards,
>    Roland
> 
>>
>> Eduard
>> -----Original Message-----
>> From: Bless, Roland (TM) <[e-mail]>
>> Sent: Friday, April 19, 2024 12:17
>> To: Vasilenko Eduard <[e-mail]>; [e-mail]
>> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
>>
>> Hi Vasilenko,
>>
>> [not answering w.r.t. the Scalable requirement, but like to discuss a
>> different point]
>>
>> I strongly disagree with your conclusion that we should forget about Window-based CC. Window-based CC has the big advantage of being self-stabilizing and thus limiting the amount of queuing delay.
>> Rate-based CCs are harder to control in this respect: the amount of queued data at the bottleneck buffer may grow over time in case you overestimated the available bandwidth.
>>
>> However, you are right that window-based CC may cause micro-bursts due
>> to various causes (application rate limits or distorted ACK clocking).
>> While I agree that the ACK clock isn't that reliable anymore these
>> days (see also
>> https://dl.acm.org/doi/10.1145/3371934.3371956 "Deprecating the TCP macroscopic model"), I disagree with abandoning window-based CCs due to that.
>> The counter-measure against micro-bursts and underutilization that may be caused by distorted ACK clocks is *pacing*!
>> This avoids micro-bursts and lets the sender keep sending for a limited time period even if ACKs are not on time.
>>
>> So my conclusion is that you should actually combine both:
>> *window-based CC + pacing* or similar to BBR rate-based sending+pacing + control of the inflight data amount (which is similar to window-based CC).
>>
>> Regards,
>>     Roland
>>
>> On 19.04.24 at 10:39 Vasilenko Eduard wrote:
>>> I see L4S as the "Congestion Control Next Generation from IETF" (that is actually in competition with "Congestion Control Next Generation from Google").
>>> Then I see the important requirement that is missing in L4S.
>>>
>>> The primary requirement for CC is that it "should not grow the buffer on the bottleneck link".
>>> It is very disputable: is "the Scalable" requirement about it or not? Let's pretend that it is about it.
>>>
>>> Then the next critical requirement is "pacing" which is missed in L4S completely.
>>> Kudos to Google because I understood its importance after one local (but big) company tested and investigated BBRv1 (then implemented it).
>>> After tests, they concluded that pacing is even more important than
>>> low latency. (I doubt, probably latency is more important) Imagine that the server would increase the window sharply. The Server may have a 100GE interface. It could generate 10us of traffic as a burst (or even more).
>>> Intermediate links could be 100GE or even bigger (highly probable), and the burst would travel as it is (without any spreading).
>>> Then this burst could arrive at 10Mbps subscriber (good performance for shared public WiFi). 0.01ms burst would become 100ms burst.
>>> It would create many negative consequences for the bottleneck link:
>>> - tail drop if buffers are not enough
>>> - guaranteed huge latency
>>> Hence, we should completely forget about W (window) while discussing CC, we have to use T (time between packets).
>>> CC next generation "should avoid bursts regulating inter-packet time, not the information permitted in transit".
>>
>