From roland.bless@kit.edu Wed Apr 24 04:19:57 2024 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4401BC15154D; Wed, 24 Apr 2024 04:19:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.094 X-Spam-Level: X-Spam-Status: No, score=-2.094 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=kit.edu Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PSNfTCZfnwk1; Wed, 24 Apr 2024 04:19:52 -0700 (PDT) Received: from scc-mailout-kit-02.scc.kit.edu (scc-mailout-kit-02.scc.kit.edu [IPv6:2a00:1398:9:f712::810d:e752]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 037FAC14F6F6; Wed, 24 Apr 2024 04:19:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kit.edu; s=kit2; h=In-Reply-To:Cc:From:References:To:Subject:MIME-Version:Date: Message-ID:Content-Type:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=LCbjYj/r+ljEscwZWbNq7UKpTzzXIIXJ0tz85WtDMEA=; b=nS1mr7uFuizTsfCHrwLfeUG32Y 7R57bTH2mRAztL0lUH4qCyLr+8988tlzKjlIXdtvCaQlK4GmEmKpaERfj0n04d+iCKuAPGs2uf6ad nfumrhciVeYPvk0a31lPbiBgMQChqPjjhAs/yc71k0HVmIwRv+aTmVdTHmCvqgwSsg3PRWdAp0Nbf GDorqMbKubQZFnQJdlDwwCG0HJCpPVB2k1FkWTesFDVvH2eUSMnkSD8YEwK9jlt74YIKz/oxX6L0D qm6sKrefE59ok2C9Yj42WwIT+9A5aA+8GHW26Ma9nN3R8t2EPY/xQPFwN80az7I3yIQfcuPdCjmOR GOVP0J3Q==; Received: from iramx1.informatik.kit.edu ([2a00:1398:2::10:80]) by scc-mailout-kit-02.scc.kit.edu with esmtps (TLS1.2:ECDHE_SECP256R1__RSA_SHA512__AES_256_GCM:256) (envelope-from ) id 1rzafC-008pTs-05; Wed, 24 Apr 2024 13:19:46 +0200 Received: from [2a00:1398:2:4006:b2:264f:9ac3:6afa] (helo=i72vorta.tm.kit.edu) by iramx1.informatik.kit.edu with esmtpsa port 25 iface 2a00:1398:2::10:8 id 1rzaf9-000000004i2-2Z0t; Wed, 24 Apr 2024 13:19:43 +0200 Received: from [IPV6:::1] (ip6-localhost [IPv6:::1]) by i72vorta.tm.kit.edu (Postfix) with ESMTPS id 512F1D00242; Wed, 24 Apr 2024 13:19:43 +0200 (CEST) Content-Type: multipart/alternative; boundary="------------WNqKPbi0bSAobvDYQ10iehNw" Message-ID: <6dcfa577-5a0b-499e-a254-726503a2b84c@kit.edu> Date: Wed, 24 Apr 2024 13:19:43 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Vasilenko Eduard , "tsvwg@ietf.org" References: <12c7c1300b004691a59ac950e66c3e2b@huawei.com> <554dc629154245489981e22463d3d1dc@huawei.com> From: "Bless, Roland (TM)" Content-Language: de-DE, en-US Autocrypt: addr=roland.bless@kit.edu; keydata= xsFNBFi0OxABEACy2VohJ7VhSu/xPCt4/6qCrw4Pw2nSklWPfAYEk1QgrbiwgvLAP9WEhAIU w45cojBaDxytIGg8eaYeIKSmsXjHGbV/ZTfo8r11LX8yPYR0WHiMWZpl0SHUd/CZIkv2pChO 88vF/2FKN95HDcp24pwONF4VhxJoSFk6c0mDNf8Em/Glt9BcWX2AAvizTmpQDshaPje18WH3 4++KwPZDd/sJ/hHSXiPg1Gdhs/OG/C0CJguOAlqbgSVAe3qKOr1M4K5M+wVpsk373pXRfxd7 ZAmZ05iBTn+LfgVcz+AfaKKcsWri5CdTT+7JDL6QNQpox+b5FXZFSHnEIST+/qzfG7G2LqqY mml6TYY8XbaNyXZP0QKncfSpRx8uTRWReHUa1YbSuOxXYh6bXpcugD25mlC/Lu0g7tz4ijiK iIwq9+P2H1KfAAfYyYZh6nOoE6ET0TjOjUSa+mA8cqjPWX99kEEgf1Xo+P9fx9QLCLWIY7zc mSM+vjQKgdUFpMSCKcYEKOuwlPuOz8bVECafxaEtJJHjCOK8zowe2eC9OM+G+bmtAO3qYcYZ hQ/PV3sztt/PjgdtnFAYPFLc9189rHRxKsWSOb4xPkRw/YQAI9l15OlUEpsyOehxmAmTsesn tSViCz++PCdeXrQc1BCgl8nDytrxW+n5w1aaE8aL3hn8M0tonQARAQABzShSb2xhbmQgQmxl c3MgKFRNKSA8cm9sYW5kLmJsZXNzQGtpdC5lZHU+wsGABBMBCAAqAhsDBQkSzAMABQsJCAcC BhUICQoLAgQWAgMBAh4BAheABQJYtYdHAhkBAAoJEKON2tlkOJXuzWkP+wfjUnDNzRm4r34a AMWepcQziTgqf4I1crcL6VD44767HhyFsjcKH31E5G5gTDxbpsM4pmkghKeLrpPo30YK3qb7 E9ifIkpJTvMu0StSUmcXq0zPyHZ+HxHeMWkosljG3g/4YekCqgWwrB62T7NMYq0ATQe1MGCZ TAPwSPGCUZT3ioq50800FMI8okkGTXS3h2U922em7k8rv7E349uydv19YEcS7tI78pggMdap ASoP3QWB03tzPKwjqQqSevy64uKDEa0UgvAM3PRbJxOYZlX1c3q/CdWwpwgUiAhMtPWvavWW Tcw6Kkk6e0gw4oFlDQ+hZooLv5rlYR3egdV4DPZ1ugL51u0wQCQG9qKIMXslAdmKbRDkEcWG Oi2bWAdYyIHhhQF5LSuaaxC2P2vOYRHnE5yv5KTV3V7piFgPFjKDW+giCRd7VGfod6DY2b2y zwidCMve1Qsm8+NErH6U+hMpMLeCJDMu1OOvXYbFnTkqjeg5sKipUoSdgXsIo4kl+oArZlpK qComSTPhij7rMyeu/1iOwbNCjtiqgb55ZE7Ekd84mr9sbq4Jm/4QGnVI30q4U2vdGSeNbVjo d1nqjf3UNzP2ZC+H9xjsCFuKYbCX6Yy4SSuEcubtdmdBqm13pxua4ZqPSI0DQST2CHC7nxL1 AaRGRYYh5zo2vRg3ipkEzsFNBFi0OxABEAC2CJNp0/Ivkv4KOiXxitsMXZeK9fI0NU2JU1rW 04dMLF63JF8AFiJ6qeSL2mPHoMiL+fG5jlxy050xMdpMKxnhDVdMxwPtMiGxbByfvrXu18/M B7h+E1DHYVRdFFPaL2jiw+Bvn6wTT31MiuG9Wh0WAhoW8jY8IXxKQrUn7QUOKsWhzNlvVpOo SjMiW4WXksUA0EQVbmlskS/MnFOgCr8q/FqwC81KPy+VLHPB9K/B65uQdpaw78fjAgQVQqpx H7gUF1EYpdZWyojN+V8HtLJx+9yWAZjSFO593OF3/r0nDHEycuOjhefCrqr0DDgTYUNthOdU KO2CzT7MtweRtAf0n27zbwoYvkTviIbR+1lV1vNkxaUtZ6e1rtOxvonRM1O3ddFIzRp/Qufu HfPe0YqhEsrBIGW1aE/pZW8khNQlB6qt20snL9cFDrnB6+8kDG3e//OjK1ICQj9Y/yyrJVaX KfPbdHhLpsgh8TMDPoH+XXQlDJljMD0++/o7ckO3Sfa8Zsyh1WabyKQDYXDmDgi9lCoaQ7Lf uLUpoMvJV+EWo0jE4RW/wBGQbLJp5usy5i0fhBKuDwsKdLG3qOCf4depIcNuja6ZmZHRT+3R FFjvZ/dAhrCWpRTxZANlWlLZz6htToJulAZQJD6lcpVr7EVgDX/y4cNwKF79egWXPDPOvQAR AQABwsFlBBgBCAAPBQJYtDsQAhsMBQkSzAMAAAoJEKON2tlkOJXukMoP/jNeiglj8fenH2We 7SJuyBp8+5L3n8eNwfwY5C5G+etD0E6/lkt/Jj9UddTazxeB154rVFXRzmcN3+hGCOZgGAyV 1N7d8xM6dBqRtHmRMPu5fUxfSqrM9pmqAw2gmzAe0eztVvaM+x5x5xID2WZOiOq8dx9KOKrp Zorekjs3GEA3V1wlZ7Nksx/o8KZ04hLeKcR1r06zEDLN/yA+Fz8IPa0KqpuhrL010bQDgAhe 9o5TA0/cMJpxpLqHhX2As+5cQAhKDDsWJu3oBzZRkN7Hh/HTpWurmTQRRniLGSeiL0zdtilX fowyxGXH6QWi3MZYmpOq+etr7o4EGGbm2inxpVbM+NYmaJs+MAi/z5bsO/rABwdM5ysm8hwb CGt+1oEMORyMcUk/uRjclgTZM1NhGoXm1Un67+Rehu04i7DA6b8dd1H8AFgZSO2H4IKi+5yA Ldmo+ftCJS83Nf6Wi6hJnKG9aWQjKL+qmZqBEct/D2uRJGWAERU5+D0RwNV/i9lQFCYNjG9X Tew0BPYYnBtHFlz9rJTqGhDu4ubulSkbxAK3TIk8XzKdMvef3tV/7mJCmcaVbJ2YoNUtkdKJ goOigJTMBXMRu4Ibyq1Ei+d90lxhojKKlf9yguzpxk5KYFGUizp0dtvdNuXRBtYrwzykS6vB zTlLqHZ0pvGjNfTSvuuN Organization: Institute of Telematics, Karlsruhe Institute of Technology (KIT) Cc: ccwg@ietf.org In-Reply-To: <554dc629154245489981e22463d3d1dc@huawei.com> X-ATIS-AV: ClamAV (iramx1.informatik.kit.edu) X-ATIS-Checksum: v3zoCAcc32ckk X-ATIS-Timestamp: iramx1.informatik.kit.edu esmtpsa 1713957583.697641704 Archived-At: Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Apr 2024 11:19:57 -0000 This is a multi-part message in MIME format. --------------WNqKPbi0bSAobvDYQ10iehNw Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi Eduard, On 23.04.24 at 13:52 Vasilenko Eduard wrote: > Thanks for the discussion. > I suspect that we do not understand each other because I am a stranger here and do not use proper terminology. > > There was a legacy modeling theory that every session should send BDP=RTT_min*Bottleneck_Bandwidth to fill the pipe. I prefer to speak of inflight data. Controlling the amount of inflight data (i.e., sent, but not yet acknowledged ) D_inflight is key to congestion control from the POV of a flow. The notion of inflight data includes the concept that you get some feedback from the receiver. ACKs will tell you how much data has left/not left the network. Typically, CWnd is an upper bound for D_inflight. You are right that D_inflight should be >=BDP as the bottleneck link capacity is used inefficiently otherwise. > BDP could be treated as "a window", measured in MBytes. > Your claim that the window is always needed is probably rooted in this fact - just because everybody is so accustomed to that legacy concept. Nope.As I said, using a CWnd (with ACK feedback) to limit the amount of inflight data has a self-stabilizing effect. > Free your mind - forget about BDP - it is just one way of modeling the session. > It was good in 1988 when SMTP or FTP was tolerant to latency. > Indeed, if you model the session in this way - you may have theoretical proof that in some situations "buffer is growing to infinity". Paced sending with a target rate for a certain time period without getting any feedback may be necessary as the ACK clock is distorted by various effects that we see in today's networks. This avoids underutilization if ACK feedback is not continuous (delayed ACKs, stretched ACKs, Offloading etc.). However, sending blindly for a too long time implies the risk of overloading the bottleneck, leading to queuing delay and packet loss. > Good CCA could predict/measure RTT_min and Bottleneck_Bandwidth separately (the last one is not even needed in some situations because it would be reflected inside the regulated parameter Inter_Packet_Interval). > The power of the new approach is in the fact that these parameters are treated separately (do not try to multiply it to get BDP/Window). > Keep source parameters as milliseconds and bits_per_second. That would correspond to a target sending rate, right? > For the case of RTT increase, Bottleneck_Bandwidth is not even needed - it is possible to just increase Inter_Packet_Interval in the same proportion as RTT increase to RTT_min. > For the case of ECN/drop, the packet size needs to be converted to the additional latency of the additional bottleneck queue (Bottleneck_Bandwidth would be needed), then this additional latency could be treated as RTT increase that should increase Inter_Packet_Interval in the same proportion as for the previous case. > Bottleneck_Bandwidth may be completely canceled because it is easy to calculate from the Inter_Packet_Interval. > Window/BDP would not appear anywhere in these calculations. I'm not sure that I understood your approach, but sure, if you modulate the sending rate according to some feedback, that may work as well, depending on the feedback loop time scale. > I do not understand why you are talking about the burst because if CCA regulates Inter_Packet_Interval then a burst is not possible by default. My case was that you get micro-bursts if you are not using paced sending. When using pacing, you can still get bursts if the allowed sending rate is somewhat larger than the ideal sending rate. > The thing opposite to the burst is possible: Bottleneck_Bandwidth may be abruptly changed (especially on wireless) which would have a similar effect to the burst. A standing queue at the bottleneck provides advantages in these cases, where the bottleneck bandwidth increases: there are packets waiting to be served already present, so utilization can be kept high. However, that comes with the usual latency trade-off. > If Bottleneck_Bandwidth has decreased then CCA would get ECN or RTT increase, > If Bottleneck_Bandwidth has increased then CCA should have a strategy to probe smaller Inter_Packet_Interval periodically. > In automation control theory, instability is possible only if the control loop is slower than the regulated system. > Then the tradeoff is possible: the accuracy of regulation to stability (do not change Inter_Packet_Interval too fast, use interpolation or averaging). My point is that modulating the sending rate may be fragile even at smaller timescales, whereas the congestion window automatically stabilizes the sending rate. For example, using a pacing rate of 120% of the ideal rate may cause infinite queue growth already. > > I need to investigate BBR details more, but I had an impression that BBR is doing something like the above. BBR basically measures the delivery rate and tries to send at a "reasonable" paced rate, while also controlling the amount of inflight data. An inflight cap is calculated based on the measured BDP. It is chosen large enough to deal with lacking ACK feedback without sacrificing bottleneck utilization. Regards,  Roland > Eduard > -----Original Message----- > From: Bless, Roland (TM) > Sent: Friday, April 19, 2024 19:07 > To: Vasilenko Eduard;tsvwg@ietf.org > Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S > > Hi Eduard, > > [sorry messed up your first name], see inline. > > On 19.04.24 at 15:16 Vasilenko Eduard wrote: >> Thanks. Looks like I have understood now. >> Pacing or rate-limiting is about what is the averaging interval. >> >> The example that I have presented initially (bump to the low-speed link) needs pacing (on the small interval). >> I still do not understand why it is so mandatory to have the "window" concept. > It is not mandatory, but useful IMHO, because window-based congestion control > - automatically achieves stability even if window size is not perfect > - implicitly adjusts the sending rate > - limits the amount of inflight data -> and thus the queue length > - if the window is too large: you get a standing queue at the bottleneck, but no instability (i.e., steadily growing queue) > >> Because if CC is capable of limiting the bottleneck queue in any way then "unlimited queue growth" is not possible. > That is correct, but the question is _how_ the bottleneck queue can be limited. Simply picking a "suitable" sending rate is not sufficient since burstiness may cause unlimited queue growth. As I said: > at a 100Mbit/s bottleneck, two senders with an average data rate of _exactly_ 50Mbit/s but exponentially distributed sending behaviors show enough burstiness to cause unlimited queue growth (this is also confirmed by queuing theory). > >> I believe that the clear ECN signal is very good for this purpose, but observing the RTT looks not bad too (as BBR has proved). > Yes, using multiple congestion signal are always good. BBR does not use the RTT as congestion signal BTW, which seems to be a common misunderstanding. BBR uses an estimation of RTT_min to calculate the BDP for a flow. BBRv2/3 use ECN and packet loss (usually with threshold > 2%) as congestion signals, but the designers deliberately do not use the measured RTT as it may include too much noise. > >> If one has any of these then one does not need a "window". >> >> "Congestion window" is a very good mechanism to get a bigger latency. > Nope, generally speaking, if you use the perfectly fitting window size there will be no queuing delay (if combined with pacing to avoid micro-bursts). > We have designed TCP-LoLa [*] that is delay- and window-based and able to limit the queuing delay to a configured bound (e.g., 5ms for all the flows at the same bottleneck). > [*]https://ieeexplore.ieee.org/document/8109356 > > Regards, > Roland > >> Anyway, the fact that L4S is not talking about how to mitigate burstiness is a big hole in the "Next Generation Congestion Control". >> Eduard >> -----Original Message----- >> From: Bless, Roland (TM) >> Sent: Friday, April 19, 2024 13:57 >> To: Vasilenko Eduard;tsvwg@ietf.org >> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S >> >> Hi Vasilenko, >> >> On 19.04.24 at 12:01 Vasilenko Eduard wrote: >>> 1. >>> CC may measure RTT, understand buffer growth, and increase the interval between packets. >> Delay-based CC is a different aspect and orthogonal to pacing, but typically pacing is also beneficial for them. >> >>> Reaction by W and T looks the same effective against loaded buffer. >>> Actially W is easy to re-calculate to T. I do not see a problem. >> As I said, the difference is that a sheer existence of a congestion window has a self-stabilizing effect on buffer occupation, whereas two senders that simply send with perfect average rate shares towards a bottleneck (but slightly bursty sending rate, e.g., packet sending times from an exponential distribution) may cause unlimited queue growth. >> >>> 2. >>> It looks like I use the wrong terminology because "pacing" for me is exactly the situation when information is sent on some timer (that may change slowly). >>> But you use "pacing" in combination with "BBR rate-based sending" which looks to me like "pacing for pacing". >>> It looks like "pacing" is something different for you. >>> Probably it is my fault. But I have not understood. >> Personally, I actually like to distinguish between rate-based sending and pacing: >> BBR calculates a target sending rate and uses paced sending to avoid burstiness even on smaller time scales. You could effectively use the same target sending rate over a larger interval, e.g., RTT_min and send it in a much more bursty manner. >> That would still be rate-based, but without pacing. >> >> Regards, >> Roland >> >>> Eduard >>> -----Original Message----- >>> From: Bless, Roland (TM) >>> Sent: Friday, April 19, 2024 12:17 >>> To: Vasilenko Eduard;tsvwg@ietf.org >>> Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S >>> >>> Hi Vasilenko, >>> >>> [not answering w.r.t. the Scalable requirement, but like to discuss a >>> different point] >>> >>> I strongly disagree with your conclusion that we should forget about Window-based CC. Window-based CC has the big advantage of being self-stabilizing and thus limiting the amount of queuing delay. >>> Rate-based CCs are harder to control in this respect: the amount of queued data at the bottleneck buffer may grow over time in case you overestimated the available bandwidth. >>> >>> However, you are right that window-based CC may cause micro-bursts >>> due to various causes (application rate limits or distorted ACK clocking). >>> While I agree that the ACK clock isn't that reliable anymore these >>> days (see also >>> https://dl.acm.org/doi/10.1145/3371934.3371956 "Deprecating the TCP macroscopic model"), I disagree with abandoning window-based CCs due to that. >>> The counter-measure against micro-bursts and underutilization that may be caused by distorted ACK clocks is *pacing*! >>> This avoids micro-bursts and lets the sender keep sending for a limited time period even if ACKs are not on time. >>> >>> So my conclusion is that you should actually combine both: >>> *window-based CC + pacing* or similar to BBR rate-based sending+pacing + control of the inflight data amount (which is similar to window-based CC). >>> >>> Regards, >>> Roland >>> >>> On 19.04.24 at 10:39 Vasilenko Eduard wrote: >>>> I see L4S as the "Congestion Control Next Generation from IETF" (that is actually in competition with "Congestion Control Next Generation from Google"). >>>> Then I see the important requirement that is missing in L4S. >>>> >>>> The primary requirement for CC is that it "should not grow the buffer on the bottleneck link". >>>> It is very disputable: is "the Scalable" requirement about it or not? Let's pretend that it is about it. >>>> >>>> Then the next critical requirement is "pacing" which is missed in L4S completely. >>>> Kudos to Google because I understood its importance after one local (but big) company tested and investigated BBRv1 (then implemented it). >>>> After tests, they concluded that pacing is even more important than >>>> low latency. (I doubt, probably latency is more important) Imagine that the server would increase the window sharply. The Server may have a 100GE interface. It could generate 10us of traffic as a burst (or even more). >>>> Intermediate links could be 100GE or even bigger (highly probable), and the burst would travel as it is (without any spreading). >>>> Then this burst could arrive at 10Mbps subscriber (good performance for shared public WiFi). 0.01ms burst would become 100ms burst. >>>> It would create many negative consequences for the bottleneck link: >>>> - tail drop if buffers are not enough >>>> - guaranteed huge latency >>>> Hence, we should completely forget about W (window) while discussing CC, we have to use T (time between packets). >>>> CC next generation "should avoid bursts regulating inter-packet time, not the information permitted in transit". --------------WNqKPbi0bSAobvDYQ10iehNw Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
Hi Eduard,

On 23.04.24 at 13:52 Vasilenko Eduard wrote:
Thanks for the discussion.
I suspect that we do not understand each other because I am a stranger here and do not use proper terminology.

There was a legacy modeling theory that every session should send BDP=RTT_min*Bottleneck_Bandwidth to fill the pipe.

I prefer to speak of inflight data. Controlling the amount of inflight data (i.e., sent, but not yet acknowledged ) D_inflight is key to
congestion control from the POV of a flow. The notion of inflight data includes the concept that you get some feedback from the receiver.
ACKs will tell you how much data has left/not left the network. Typically, CWnd is an upper bound for D_inflight. You are right that D_inflight
should be >=BDP as the bottleneck link capacity is used inefficiently otherwise. 

BDP could be treated as "a window", measured in MBytes.
Your claim that the window is always needed is probably rooted in this fact - just because everybody is so accustomed to that legacy concept.
Nope. As I said, using a CWnd (with ACK feedback) to limit the amount of inflight data has a self-stabilizing effect. 
Free your mind - forget about BDP - it is just one way of modeling the session.
It was good in 1988 when SMTP or FTP was tolerant to latency.
Indeed, if you model the session in this way - you may have theoretical proof that in some situations "buffer is growing to infinity".
Paced sending with a target rate for a certain time period without getting any feedback
may be necessary as the ACK clock is distorted by various effects that we see in today's networks.
This avoids underutilization if ACK feedback is not continuous (delayed ACKs, stretched ACKs, Offloading etc.).
However, sending blindly for a too long time implies the risk of overloading the bottleneck, leading
to queuing delay and packet loss. 
Good CCA could predict/measure RTT_min and Bottleneck_Bandwidth separately (the last one is not even needed in some situations because it would be reflected inside the regulated parameter Inter_Packet_Interval).
The power of the new approach is in the fact that these parameters are treated separately (do not try to multiply it to get BDP/Window).
Keep source parameters as milliseconds and bits_per_second.
That would correspond to a target sending rate, right? 
For the case of RTT increase, Bottleneck_Bandwidth is not even needed - it is possible to just increase Inter_Packet_Interval in the same proportion as RTT increase to RTT_min.
For the case of ECN/drop, the packet size needs to be converted to the additional latency of the additional bottleneck queue (Bottleneck_Bandwidth would be needed), then this additional latency could be treated as RTT increase that should increase Inter_Packet_Interval in the same proportion as for the previous case.
Bottleneck_Bandwidth may be completely canceled because it is easy to calculate from the Inter_Packet_Interval.
Window/BDP would not appear anywhere in these calculations.
I'm not sure that I understood your approach, but sure, if you modulate the sending rate according to some feedback, that may work as well, depending on the feedback loop time scale. 
I do not understand why you are talking about the burst because if CCA regulates Inter_Packet_Interval then a burst is not possible by default.
My case was that you get micro-bursts if you are not using paced sending. When using pacing, you can still get bursts if the allowed sending rate is somewhat larger
than the ideal sending rate.
The thing opposite to the burst is possible: Bottleneck_Bandwidth may be abruptly changed (especially on wireless) which would have a similar effect to the burst.
A standing queue at the bottleneck provides advantages in these cases, where the bottleneck bandwidth increases:
there are packets waiting to be served already present, so utilization can be kept high. However, that comes with
the usual latency trade-off.
If Bottleneck_Bandwidth has decreased then CCA would get ECN or RTT increase,
If Bottleneck_Bandwidth has increased then CCA should have a strategy to probe smaller Inter_Packet_Interval periodically.
In automation control theory, instability is possible only if the control loop is slower than the regulated system.
Then the tradeoff is possible: the accuracy of regulation to stability (do not change Inter_Packet_Interval too fast, use interpolation or averaging).
My point is that modulating the sending rate may be fragile even at smaller timescales, whereas the congestion window automatically stabilizes the sending rate.
For example, using a pacing rate of 120% of the ideal rate may cause infinite queue growth already.

I need to investigate BBR details more, but I had an impression that BBR is doing something like the above.
BBR basically measures the delivery rate and tries to send at a "reasonable" paced rate, while also controlling
the amount of inflight data. An inflight cap is calculated based on the measured BDP. It is chosen large enough
to deal with lacking ACK feedback without sacrificing bottleneck utilization.

Regards,
 Roland

Eduard
-----Original Message-----
From: Bless, Roland (TM) <roland.bless@kit.edu> 
Sent: Friday, April 19, 2024 19:07
To: Vasilenko Eduard <vasilenko.eduard@huawei.com>; tsvwg@ietf.org
Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S

Hi Eduard,

[sorry messed up your first name], see inline.

On 19.04.24 at 15:16 Vasilenko Eduard wrote:

Thanks. Looks like I have understood now.
Pacing or rate-limiting is about what is the averaging interval.

The example that I have presented initially (bump to the low-speed link) needs pacing (on the small interval).


      

        
I still do not understand why it is so mandatory to have the "window" concept.


      

      
It is not mandatory, but useful IMHO, because window-based congestion control
  - automatically achieves stability even if window size is not perfect
  - implicitly adjusts the sending rate
  - limits the amount of inflight data -> and thus the queue length
  - if the window is too large: you get a standing queue at the bottleneck, but no instability (i.e., steadily growing queue)



      

        
Because if CC is capable of limiting the bottleneck queue in any way then "unlimited queue growth" is not possible.


      

      
That is correct, but the question is _how_ the bottleneck queue can be limited. Simply picking a "suitable" sending rate is not sufficient since burstiness may cause unlimited queue growth. As I said:
at a 100Mbit/s bottleneck, two senders with an average data rate of _exactly_ 50Mbit/s but exponentially distributed sending behaviors show enough burstiness to cause unlimited queue growth (this is also confirmed by queuing theory).



      

        
I believe that the clear ECN signal is very good for this purpose, but observing the RTT looks not bad too (as BBR has proved).


      

      
Yes, using multiple congestion signal are always good. BBR does not use the RTT as congestion signal BTW, which seems to be a common misunderstanding. BBR uses an estimation of RTT_min to calculate the BDP for a flow. BBRv2/3 use ECN and packet loss (usually with threshold
2%) as congestion signals, but the designers deliberately do not use the measured RTT as it may include too much noise.



      

        
If one has any of these then one does not need a "window".

"Congestion window" is a very good mechanism to get a bigger latency.


      

      
Nope, generally speaking, if you use the perfectly fitting window size there will be no queuing delay (if combined with pacing to avoid micro-bursts).
We have designed TCP-LoLa [*] that is delay- and window-based and able to limit the queuing delay to a configured bound (e.g., 5ms for all the flows at the same bottleneck).
[*] https://ieeexplore.ieee.org/document/8109356

Regards,
  Roland



      

        
Anyway, the fact that L4S is not talking about how to mitigate burstiness is a big hole in the "Next Generation Congestion Control".
Eduard
-----Original Message-----
From: Bless, Roland (TM) <roland.bless@kit.edu>
Sent: Friday, April 19, 2024 13:57
To: Vasilenko Eduard <vasilenko.eduard@huawei.com>; tsvwg@ietf.org
Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S

Hi Vasilenko,

On 19.04.24 at 12:01 Vasilenko Eduard wrote:


        

          
1.
CC may measure RTT, understand buffer growth, and increase the interval between packets.


        

        
Delay-based CC is a different aspect and orthogonal to pacing, but typically pacing is also beneficial for them.



        

          
Reaction by W and T looks the same effective against loaded buffer.
Actially W is easy to re-calculate to T. I do not see a problem.


        

        
As I said, the difference is that a sheer existence of a congestion window has a self-stabilizing effect on buffer occupation, whereas two senders that simply send with perfect average rate shares towards a bottleneck (but slightly bursty sending rate, e.g., packet sending times from an exponential distribution) may cause unlimited queue growth.



        

          
2.
It looks like I use the wrong terminology because "pacing" for me is exactly the situation when information is sent on some timer (that may change slowly).
But you use "pacing" in combination with "BBR rate-based sending" which looks to me like "pacing for pacing".
It looks like "pacing" is something different for you.
Probably it is my fault. But I have not understood.


        

        
Personally, I actually like to distinguish between rate-based sending and pacing:
BBR calculates a target sending rate and uses paced sending to avoid burstiness even on smaller time scales. You could effectively use the same target sending rate over a larger interval, e.g., RTT_min and send it in a much more bursty manner.
That would still be rate-based, but without pacing.

Regards,
   Roland



        

          
Eduard
-----Original Message-----
From: Bless, Roland (TM) <roland.bless@kit.edu>
Sent: Friday, April 19, 2024 12:17
To: Vasilenko Eduard <vasilenko.eduard@huawei.com>; tsvwg@ietf.org
Subject: Re: [tsvwg] "Pacing" requirement is lost in L4S

Hi Vasilenko,

[not answering w.r.t. the Scalable requirement, but like to discuss a 
different point]

I strongly disagree with your conclusion that we should forget about Window-based CC. Window-based CC has the big advantage of being self-stabilizing and thus limiting the amount of queuing delay.
Rate-based CCs are harder to control in this respect: the amount of queued data at the bottleneck buffer may grow over time in case you overestimated the available bandwidth.

However, you are right that window-based CC may cause micro-bursts 
due to various causes (application rate limits or distorted ACK clocking).
While I agree that the ACK clock isn't that reliable anymore these 
days (see also
https://dl.acm.org/doi/10.1145/3371934.3371956 "Deprecating the TCP macroscopic model"), I disagree with abandoning window-based CCs due to that.
The counter-measure against micro-bursts and underutilization that may be caused by distorted ACK clocks is *pacing*!
This avoids micro-bursts and lets the sender keep sending for a limited time period even if ACKs are not on time.

So my conclusion is that you should actually combine both:
*window-based CC + pacing* or similar to BBR rate-based sending+pacing + control of the inflight data amount (which is similar to window-based CC).

Regards,
    Roland

On 19.04.24 at 10:39 Vasilenko Eduard wrote:


          

            
I see L4S as the "Congestion Control Next Generation from IETF" (that is actually in competition with "Congestion Control Next Generation from Google").
Then I see the important requirement that is missing in L4S.

The primary requirement for CC is that it "should not grow the buffer on the bottleneck link".
It is very disputable: is "the Scalable" requirement about it or not? Let's pretend that it is about it.

Then the next critical requirement is "pacing" which is missed in L4S completely.
Kudos to Google because I understood its importance after one local (but big) company tested and investigated BBRv1 (then implemented it).
After tests, they concluded that pacing is even more important than 
low latency. (I doubt, probably latency is more important) Imagine that the server would increase the window sharply. The Server may have a 100GE interface. It could generate 10us of traffic as a burst (or even more).
Intermediate links could be 100GE or even bigger (highly probable), and the burst would travel as it is (without any spreading).
Then this burst could arrive at 10Mbps subscriber (good performance for shared public WiFi). 0.01ms burst would become 100ms burst.
It would create many negative consequences for the bottleneck link:
- tail drop if buffers are not enough
- guaranteed huge latency
Hence, we should completely forget about W (window) while discussing CC, we have to use T (time between packets).
CC next generation "should avoid bursts regulating inter-packet time, not the information permitted in transit".


          

          

        

        

      

      

    

    


    

  


--------------WNqKPbi0bSAobvDYQ10iehNw--