Re: [tsvwg] "Pacing" requirement is lost in L4S

["Bless, Roland (TM)" <[e-mail]>]

Hi Eduard,

On 23.04.24 at 13:52 Vasilenko Eduard wrote:
> Thanks for the discussion.
> I suspect that we do not understand each other because I am a stranger here and do not use proper terminology.
>
> There was a legacy modeling theory that every session should send BDP=RTT_min*Bottleneck_Bandwidth to fill the pipe.

I prefer to speak of inflight data. Controlling the amount of inflight 
data (i.e., sent, but not yet acknowledged ) D_inflight is key to
congestion control from the POV of a flow. The notion of inflight data 
includes the concept that you get some feedback from the receiver.
ACKs will tell you how much data has left/not left the network. 
Typically, CWnd is an upper bound for D_inflight. You are right that 
D_inflight
should be >=BDP as the bottleneck link capacity is used inefficiently 
otherwise.

> BDP could be treated as "a window", measured in MBytes.
> Your claim that the window is always needed is probably rooted in this fact - just because everybody is so accustomed to that legacy concept.
Nope.As I said, using a CWnd (with ACK feedback) to limit the amount of 
inflight data has a self-stabilizing effect.
> Free your mind - forget about BDP - it is just one way of modeling the session.
> It was good in 1988 when SMTP or FTP was tolerant to latency.
> Indeed, if you model the session in this way - you may have theoretical proof that in some situations "buffer is growing to infinity".
Paced sending with a target rate for a certain time period without 
getting any feedback
may be necessary as the ACK clock is distorted by various effects that 
we see in today's networks.
This avoids underutilization if ACK feedback is not continuous (delayed 
ACKs, stretched ACKs, Offloading etc.).
However, sending blindly for a too long time implies the risk of 
overloading the bottleneck, leading
to queuing delay and packet loss.
> Good CCA could predict/measure RTT_min and Bottleneck_Bandwidth separately (the last one is not even needed in some situations because it would be reflected inside the regulated parameter Inter_Packet_Interval).
> The power of the new approach is in the fact that these parameters are treated separately (do not try to multiply it to get BDP/Window).
> Keep source parameters as milliseconds and bits_per_second.
That would correspond to a target sending rate, right?
> For the case of RTT increase, Bottleneck_Bandwidth is not even needed - it is possible to just increase Inter_Packet_Interval in the same proportion as RTT increase to RTT_min.
> For the case of ECN/drop, the packet size needs to be converted to the additional latency of the additional bottleneck queue (Bottleneck_Bandwidth would be needed), then this additional latency could be treated as RTT increase that should increase Inter_Packet_Interval in the same proportion as for the previous case.
> Bottleneck_Bandwidth may be completely canceled because it is easy to calculate from the Inter_Packet_Interval.
> Window/BDP would not appear anywhere in these calculations.
I'm not sure that I understood your approach, but sure, if you modulate 
the sending rate according to some feedback, that may work as well, 
depending on the feedback loop time scale.
> I do not understand why you are talking about the burst because if CCA regulates Inter_Packet_Interval then a burst is not possible by default.
My case was that you get micro-bursts if you are not using paced 
sending. When using pacing, you can still get bursts if the allowed 
sending rate is somewhat larger
than the ideal sending rate.
> The thing opposite to the burst is possible: Bottleneck_Bandwidth may be abruptly changed (especially on wireless) which would have a similar effect to the burst.
A standing queue at the bottleneck provides advantages in these cases, 
where the bottleneck bandwidth increases:
there are packets waiting to be served already present, so utilization 
can be kept high. However, that comes with
the usual latency trade-off.
> If Bottleneck_Bandwidth has decreased then CCA would get ECN or RTT increase,
> If Bottleneck_Bandwidth has increased then CCA should have a strategy to probe smaller Inter_Packet_Interval periodically.
> In automation control theory, instability is possible only if the control loop is slower than the regulated system.
> Then the tradeoff is possible: the accuracy of regulation to stability (do not change Inter_Packet_Interval too fast, use interpolation or averaging).
My point is that modulating the sending rate may be fragile even at 
smaller timescales, whereas the congestion window automatically 
stabilizes the sending rate.
For example, using a pacing rate of 120% of the ideal rate may cause 
infinite queue growth already.
>
> I need to investigate BBR details more, but I had an impression that BBR is doing something like the above.
BBR basically measures the delivery rate and tries to send at a 
"reasonable" paced rate, while also controlling
the amount of inflight data. An inflight cap is calculated based on the 
measured BDP. It is chosen large enough
to deal with lacking ACK feedback without sacrificing bottleneck 
utilization.

Regards,
  Roland

> Eduard
> -----Original Message-----
> From: Bless, Roland (TM)<[e-mail]>  
> Sent: Friday, April 19, 2024 19:07
> To: Vasilenko Eduard<[e-mail]>;[e-mail]
> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
>
> Hi Eduard,
>
> [sorry messed up your first name], see inline.
>
> On 19.04.24 at 15:16 Vasilenko Eduard wrote:
>> Thanks. Looks like I have understood now.
>> Pacing or rate-limiting is about what is the averaging interval.
>>
>> The example that I have presented initially (bump to the low-speed link) needs pacing (on the small interval).
>> I still do not understand why it is so mandatory to have the "window" concept.
> It is not mandatory, but useful IMHO, because window-based congestion control
>    - automatically achieves stability even if window size is not perfect
>    - implicitly adjusts the sending rate
>    - limits the amount of inflight data -> and thus the queue length
>    - if the window is too large: you get a standing queue at the bottleneck, but no instability (i.e., steadily growing queue)
>
>> Because if CC is capable of limiting the bottleneck queue in any way then "unlimited queue growth" is not possible.
> That is correct, but the question is _how_ the bottleneck queue can be limited. Simply picking a "suitable" sending rate is not sufficient since burstiness may cause unlimited queue growth. As I said:
> at a 100Mbit/s bottleneck, two senders with an average data rate of _exactly_ 50Mbit/s but exponentially distributed sending behaviors show enough burstiness to cause unlimited queue growth (this is also confirmed by queuing theory).
>
>> I believe that the clear ECN signal is very good for this purpose, but observing the RTT looks not bad too (as BBR has proved).
> Yes, using multiple congestion signal are always good. BBR does not use the RTT as congestion signal BTW, which seems to be a common misunderstanding. BBR uses an estimation of RTT_min to calculate the BDP for a flow. BBRv2/3 use ECN and packet loss (usually with threshold
> 2%) as congestion signals, but the designers deliberately do not use the measured RTT as it may include too much noise.
>
>> If one has any of these then one does not need a "window".
>>
>> "Congestion window" is a very good mechanism to get a bigger latency.
> Nope, generally speaking, if you use the perfectly fitting window size there will be no queuing delay (if combined with pacing to avoid micro-bursts).
> We have designed TCP-LoLa [*] that is delay- and window-based and able to limit the queuing delay to a configured bound (e.g., 5ms for all the flows at the same bottleneck).
> [*]https://ieeexplore.ieee.org/document/8109356
>
> Regards,
>    Roland
>
>> Anyway, the fact that L4S is not talking about how to mitigate burstiness is a big hole in the "Next Generation Congestion Control".
>> Eduard
>> -----Original Message-----
>> From: Bless, Roland (TM)<[e-mail]>
>> Sent: Friday, April 19, 2024 13:57
>> To: Vasilenko Eduard<[e-mail]>;[e-mail]
>> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
>>
>> Hi Vasilenko,
>>
>> On 19.04.24 at 12:01 Vasilenko Eduard wrote:
>>> 1.
>>> CC may measure RTT, understand buffer growth, and increase the interval between packets.
>> Delay-based CC is a different aspect and orthogonal to pacing, but typically pacing is also beneficial for them.
>>
>>> Reaction by W and T looks the same effective against loaded buffer.
>>> Actially W is easy to re-calculate to T. I do not see a problem.
>> As I said, the difference is that a sheer existence of a congestion window has a self-stabilizing effect on buffer occupation, whereas two senders that simply send with perfect average rate shares towards a bottleneck (but slightly bursty sending rate, e.g., packet sending times from an exponential distribution) may cause unlimited queue growth.
>>
>>> 2.
>>> It looks like I use the wrong terminology because "pacing" for me is exactly the situation when information is sent on some timer (that may change slowly).
>>> But you use "pacing" in combination with "BBR rate-based sending" which looks to me like "pacing for pacing".
>>> It looks like "pacing" is something different for you.
>>> Probably it is my fault. But I have not understood.
>> Personally, I actually like to distinguish between rate-based sending and pacing:
>> BBR calculates a target sending rate and uses paced sending to avoid burstiness even on smaller time scales. You could effectively use the same target sending rate over a larger interval, e.g., RTT_min and send it in a much more bursty manner.
>> That would still be rate-based, but without pacing.
>>
>> Regards,
>>     Roland
>>
>>> Eduard
>>> -----Original Message-----
>>> From: Bless, Roland (TM)<[e-mail]>
>>> Sent: Friday, April 19, 2024 12:17
>>> To: Vasilenko Eduard<[e-mail]>;[e-mail]
>>> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
>>>
>>> Hi Vasilenko,
>>>
>>> [not answering w.r.t. the Scalable requirement, but like to discuss a
>>> different point]
>>>
>>> I strongly disagree with your conclusion that we should forget about Window-based CC. Window-based CC has the big advantage of being self-stabilizing and thus limiting the amount of queuing delay.
>>> Rate-based CCs are harder to control in this respect: the amount of queued data at the bottleneck buffer may grow over time in case you overestimated the available bandwidth.
>>>
>>> However, you are right that window-based CC may cause micro-bursts
>>> due to various causes (application rate limits or distorted ACK clocking).
>>> While I agree that the ACK clock isn't that reliable anymore these
>>> days (see also
>>> https://dl.acm.org/doi/10.1145/3371934.3371956  "Deprecating the TCP macroscopic model"), I disagree with abandoning window-based CCs due to that.
>>> The counter-measure against micro-bursts and underutilization that may be caused by distorted ACK clocks is *pacing*!
>>> This avoids micro-bursts and lets the sender keep sending for a limited time period even if ACKs are not on time.
>>>
>>> So my conclusion is that you should actually combine both:
>>> *window-based CC + pacing* or similar to BBR rate-based sending+pacing + control of the inflight data amount (which is similar to window-based CC).
>>>
>>> Regards,
>>>      Roland
>>>
>>> On 19.04.24 at 10:39 Vasilenko Eduard wrote:
>>>> I see L4S as the "Congestion Control Next Generation from IETF" (that is actually in competition with "Congestion Control Next Generation from Google").
>>>> Then I see the important requirement that is missing in L4S.
>>>>
>>>> The primary requirement for CC is that it "should not grow the buffer on the bottleneck link".
>>>> It is very disputable: is "the Scalable" requirement about it or not? Let's pretend that it is about it.
>>>>
>>>> Then the next critical requirement is "pacing" which is missed in L4S completely.
>>>> Kudos to Google because I understood its importance after one local (but big) company tested and investigated BBRv1 (then implemented it).
>>>> After tests, they concluded that pacing is even more important than
>>>> low latency. (I doubt, probably latency is more important) Imagine that the server would increase the window sharply. The Server may have a 100GE interface. It could generate 10us of traffic as a burst (or even more).
>>>> Intermediate links could be 100GE or even bigger (highly probable), and the burst would travel as it is (without any spreading).
>>>> Then this burst could arrive at 10Mbps subscriber (good performance for shared public WiFi). 0.01ms burst would become 100ms burst.
>>>> It would create many negative consequences for the bottleneck link:
>>>> - tail drop if buffers are not enough
>>>> - guaranteed huge latency
>>>> Hence, we should completely forget about W (window) while discussing CC, we have to use T (time between packets).
>>>> CC next generation "should avoid bursts regulating inter-packet time, not the information permitted in transit".