Re: [tsvwg] "Pacing" requirement is lost in L4S

[Ingemar Johansson S <[e-mail]>]

Hi

It is possible that this discussion may fit better in the CCWG, so I add that maillist too.

Question to chairs : Where do these topics fit best?. There is some overlap between tsvwg, ccwg (and iccrg), on top of that there is an l4s-discuss list and it is not perfectly clear different topics fit best and to me this thread fits quite well in all mail lists :-)

The SCReAM algorithm implements packet pacing and a relaxed use for the congestion window
https://datatracker.ietf.org/doc/draft-johansson-ccwg-rfc8298bis-screamv2/
Packet pacing is set to 50% higher than the nominal media bitrate, the reason here is to be able to push larger video frames into the network before a new video frame is ready for transmission.
The congestion window is also used in a relaxed manner and allows bytes in flight to exceed it by 50%. This again avoids that larger frames are held for transmission unnecessarily. At the same time the calcutation of the congestion window is adjusted based on one-way delay, ECN/L4S and packet loss to avoid excessive queue build-up.

This arrangement is made to handle rate adaptive video with all their features and it appears to work quite OK over both simulated 5G as well as real live network testing.

/Ingemar



PS: I got some feedback at the latest CCWG session that my use for the congestion window may be confusing as it does not impose a strict limit to bytes in flight. Later updates will use the term reference window (refWnd) instead.

> -----Original Message-----
> From: tsvwg <[e-mail]> On Behalf Of Bless, Roland (TM)
> Sent: Friday, 19 April 2024 18:07
> To: Vasilenko Eduard <[e-mail]>; [e-mail]
> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
>
> Hi Eduard,
>
> [sorry messed up your first name], see inline.
>
> On 19.04.24 at 15:16 Vasilenko Eduard wrote:
> > Thanks. Looks like I have understood now.
> > Pacing or rate-limiting is about what is the averaging interval.
> >
> > The example that I have presented initially (bump to the low-speed
> link) needs pacing (on the small interval).
>
> > I still do not understand why it is so mandatory to have the "window"
> concept.
>
> It is not mandatory, but useful IMHO, because window-based congestion
> control
>   - automatically achieves stability even if window size is not perfect
>   - implicitly adjusts the sending rate
>   - limits the amount of inflight data -> and thus the queue length
>   - if the window is too large: you get a standing queue at the
> bottleneck, but no instability (i.e., steadily growing queue)
>
> > Because if CC is capable of limiting the bottleneck queue in any way
> then "unlimited queue growth" is not possible.
>
> That is correct, but the question is _how_ the bottleneck queue can be
> limited. Simply picking a "suitable" sending rate is not sufficient
> since burstiness may cause unlimited queue growth. As I said:
> at a 100Mbit/s bottleneck, two senders with an average data rate of
> _exactly_ 50Mbit/s but exponentially distributed sending behaviors show
> enough burstiness to cause unlimited queue growth (this is also
> confirmed by queuing theory).
>
> > I believe that the clear ECN signal is very good for this purpose,
> but observing the RTT looks not bad too (as BBR has proved).
>
> Yes, using multiple congestion signal are always good. BBR does not use
> the RTT as congestion signal BTW, which seems to be a common
> misunderstanding. BBR uses an estimation of RTT_min to calculate the
> BDP for a flow. BBRv2/3 use ECN and packet loss (usually with threshold
> 2%) as congestion signals, but the designers deliberately do not use
> the measured RTT as it may include too much noise.
>
> > If one has any of these then one does not need a "window".
> >
> > "Congestion window" is a very good mechanism to get a bigger latency.
>
> Nope, generally speaking, if you use the perfectly fitting window size
> there will be no queuing delay (if combined with pacing to avoid micro-
> bursts).
> We have designed TCP-LoLa [*] that is delay- and window-based and able
> to limit the queuing delay to a configured bound (e.g., 5ms for all the
> flows at the same bottleneck).
> [*]
> https://ieeex/
> plore.ieee.org%2Fdocument%2F8109356&data=05%7C02%7Cingemar.s.johansson%
> 40ericsson.com%7C759b92c05b534d92ca9208dc608ac743%7C92e84cebfbfd47abbe5
> 2080c6b87953f%7C0%7C0%7C638491396379960343%7CUnknown%7CTWFpbGZsb3d8eyJW
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7
> C%7C&sdata=68SvGSe7ExGOEt%2Bxqrp%2FGYvpsG19J9LFktNDtLmTP5o%3D&reserved=
> 0
>
> Regards,
>   Roland
>
> > Anyway, the fact that L4S is not talking about how to mitigate
> burstiness is a big hole in the "Next Generation Congestion Control".
> > Eduard
> > -----Original Message-----
> > From: Bless, Roland (TM) <[e-mail]>
> > Sent: Friday, April 19, 2024 13:57
> > To: Vasilenko Eduard <[e-mail]>; [e-mail]
> > Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
> >
> > Hi Vasilenko,
> >
> > On 19.04.24 at 12:01 Vasilenko Eduard wrote:
> >> 1.
> >> CC may measure RTT, understand buffer growth, and increase the
> interval between packets.
> >
> > Delay-based CC is a different aspect and orthogonal to pacing, but
> typically pacing is also beneficial for them.
> >
> >> Reaction by W and T looks the same effective against loaded buffer.
> >> Actially W is easy to re-calculate to T. I do not see a problem.
> >
> > As I said, the difference is that a sheer existence of a congestion
> window has a self-stabilizing effect on buffer occupation, whereas two
> senders that simply send with perfect average rate shares towards a
> bottleneck (but slightly bursty sending rate, e.g., packet sending
> times from an exponential distribution) may cause unlimited queue
> growth.
> >
> >> 2.
> >> It looks like I use the wrong terminology because "pacing" for me is
> exactly the situation when information is sent on some timer (that may
> change slowly).
> >> But you use "pacing" in combination with "BBR rate-based sending"
> which looks to me like "pacing for pacing".
> >> It looks like "pacing" is something different for you.
> >> Probably it is my fault. But I have not understood.
> >
> > Personally, I actually like to distinguish between rate-based sending
> and pacing:
> > BBR calculates a target sending rate and uses paced sending to avoid
> burstiness even on smaller time scales. You could effectively use the
> same target sending rate over a larger interval, e.g., RTT_min and send
> it in a much more bursty manner.
> > That would still be rate-based, but without pacing.
> >
> > Regards,
> >    Roland
> >
> >>
> >> Eduard
> >> -----Original Message-----
> >> From: Bless, Roland (TM) <[e-mail]>
> >> Sent: Friday, April 19, 2024 12:17
> >> To: Vasilenko Eduard <[e-mail]>; [e-mail]
> >> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S
> >>
> >> Hi Vasilenko,
> >>
> >> [not answering w.r.t. the Scalable requirement, but like to discuss
> a
> >> different point]
> >>
> >> I strongly disagree with your conclusion that we should forget about
> Window-based CC. Window-based CC has the big advantage of being self-
> stabilizing and thus limiting the amount of queuing delay.
> >> Rate-based CCs are harder to control in this respect: the amount of
> queued data at the bottleneck buffer may grow over time in case you
> overestimated the available bandwidth.
> >>
> >> However, you are right that window-based CC may cause micro-bursts
> >> due to various causes (application rate limits or distorted ACK
> clocking).
> >> While I agree that the ACK clock isn't that reliable anymore these
> >> days (see also
> >>
> https://dl.ac/
> m.org%2Fdoi%2F10.1145%2F3371934.3371956&data=05%7C02%7Cingemar.s.johans
> son%40ericsson.com%7C759b92c05b534d92ca9208dc608ac743%7C92e84cebfbfd47a
> bbe52080c6b87953f%7C0%7C0%7C638491396379968059%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%
> 7C%7C%7C&sdata=tcgIRKPhHCvS6oPzNxh7ooNaiEgHOi8If44jrngtlQs%3D&reserved=
> 0 "Deprecating the TCP macroscopic model"), I disagree with abandoning
> window-based CCs due to that.
> >> The counter-measure against micro-bursts and underutilization that
> may be caused by distorted ACK clocks is *pacing*!
> >> This avoids micro-bursts and lets the sender keep sending for a
> limited time period even if ACKs are not on time.
> >>
> >> So my conclusion is that you should actually combine both:
> >> *window-based CC + pacing* or similar to BBR rate-based
> sending+pacing + control of the inflight data amount (which is similar
> to window-based CC).
> >>
> >> Regards,
> >>     Roland
> >>
> >> On 19.04.24 at 10:39 Vasilenko Eduard wrote:
> >>> I see L4S as the "Congestion Control Next Generation from IETF"
> (that is actually in competition with "Congestion Control Next
> Generation from Google").
> >>> Then I see the important requirement that is missing in L4S.
> >>>
> >>> The primary requirement for CC is that it "should not grow the
> buffer on the bottleneck link".
> >>> It is very disputable: is "the Scalable" requirement about it or
> not? Let's pretend that it is about it.
> >>>
> >>> Then the next critical requirement is "pacing" which is missed in
> L4S completely.
> >>> Kudos to Google because I understood its importance after one local
> (but big) company tested and investigated BBRv1 (then implemented it).
> >>> After tests, they concluded that pacing is even more important than
> >>> low latency. (I doubt, probably latency is more important) Imagine
> that the server would increase the window sharply. The Server may have
> a 100GE interface. It could generate 10us of traffic as a burst (or
> even more).
> >>> Intermediate links could be 100GE or even bigger (highly probable),
> and the burst would travel as it is (without any spreading).
> >>> Then this burst could arrive at 10Mbps subscriber (good performance
> for shared public WiFi). 0.01ms burst would become 100ms burst.
> >>> It would create many negative consequences for the bottleneck link:
> >>> - tail drop if buffers are not enough
> >>> - guaranteed huge latency
> >>> Hence, we should completely forget about W (window) while
> discussing CC, we have to use T (time between packets).
> >>> CC next generation "should avoid bursts regulating inter-packet
> time, not the information permitted in transit".
> >>
> >