[tcpm] Re: About growing cwnd when the sender is rate-limited

Neal Cardwell <ncardwell@google.com> Sat, 20 July 2024 16:05 UTC

Return-Path: <ncardwell@google.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 69975C14F69F for <tcpm@ietfa.amsl.com>; Sat, 20 Jul 2024 09:05:38 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -22.609
X-Spam-Level:
X-Spam-Status: No, score=-22.609 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FGUX1_2pYjnI for <tcpm@ietfa.amsl.com>; Sat, 20 Jul 2024 09:05:37 -0700 (PDT)
Received: from mail-oa1-x36.google.com (mail-oa1-x36.google.com [IPv6:2001:4860:4864:20::36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6D562C14F689 for <tcpm@ietf.org>; Sat, 20 Jul 2024 09:05:37 -0700 (PDT)
Received: by mail-oa1-x36.google.com with SMTP id 586e51a60fabf-260f1664fdfso1408984fac.1 for <tcpm@ietf.org>; Sat, 20 Jul 2024 09:05:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721491536; x=1722096336; darn=ietf.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=JnXUSUASMZPQDzuTxuD4sFrgOjW1vNcuYk/tFi8OnIQ=; b=f0Ppiw79/c2Gg3ciRpohpA1DOZ9jvn1Wy0wS2EZQoQwFvgafR8HK3OHVC6P61x9b1s S/JyUOtlqgEYuv3YMVrfpSGPVpdpbtEpnH/MvsznwlYFyO088Rc03i/M8QfxJPuv9jRn 9Sy4rEepCY7OTdD+eKheQ71fgPw7YTHAXbUJP4VV1+o1O3B5+uTg5qbyUsrqZv+OnUSV +AuSZ3CBCLNJ4lu9UCHI3YEPHEFpEatZByoUZ6UZblxTM0M37CyY14tIylrphA/yCAvA MhpDrHVGa1CRFjRg0HO8CmMRfvgexi/Dyc9+ptBtOSydSoweS7bEaOhuo3by4vS9oavd e1cw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721491536; x=1722096336; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JnXUSUASMZPQDzuTxuD4sFrgOjW1vNcuYk/tFi8OnIQ=; b=Q8OMQW1c4+ggM3atTK0Cjwnk9qtpRcskKxmfpIbzEQTA0uBqCW6CNkE6vq56WTuC/J Z1lMUtU3QKUcJZGayFFm23hkj3toT0VIa/flsy2jLKJIzcJO7savKQsNTL4+n5z8e5+L dicryD+lWuvA/YdPGnPdc50tZUF4JsQtlghXqk52lPq6yIzaS6TyOewlkGkQQrhykZx3 GDkOpv1K1TEkeKea6v2xdg38+EBTak5RPQLB8y1TAAGPgeN65wEwvA1MzHwuO1YwTm4H am7pmZX261dxWqhyYURSy8GIniVu7z6RNmjGvSTnIV/VmhsQX7e2sPBb05X8MayKjkQo JcWA==
X-Forwarded-Encrypted: i=1; AJvYcCXHkTJ9UmYsF4ZE1uymXOrZxBL4gib4GBQ/kfa7tUhYRBUQr4aTkO29NqLhpfmNSwYOgAOczX6Pw9xXgXXh
X-Gm-Message-State: AOJu0Yyl9IIzeAryZRdmfNo3SFRGgEd2izuOJO3UE8sPxJ6UMArY+Do5 xvPKCut5nghPK+NtBzePzKIsSZZDherkKAeTkO97Jgz8qiIj5OxthgLSeQ/J4dyJcHai0qX4zYQ D9WKMU5EZ0RE73S5MkKzXoiQNCsyKhxP/3RQJ
X-Google-Smtp-Source: AGHT+IHZupzKMx/bmcoLr9hJYgD2AraqTXTuMh4r8lzyZEPFS7KKierhcspOkxjBL7+b/5tBHdbP4mgExq1InmlHLAs=
X-Received: by 2002:a05:6871:b12:b0:263:3b45:b7dd with SMTP id 586e51a60fabf-2638df809ffmr1273448fac.1.1721491536005; Sat, 20 Jul 2024 09:05:36 -0700 (PDT)
MIME-Version: 1.0
References: <170896098131.16189.4842811868600508870@ietfa.amsl.com> <CADVnQy=rvCoQC0RwVq=P2XWFGPrXvGKvj2cAooj94yx+WzXz3A@mail.gmail.com> <8e5f0a7-b39b-cfaa-5c38-edeb9916bef6@cs.helsinki.fi> <CADVnQynR99fQjWmYj-rYZ4nZxYS=-O7zbfWjJLMxd5Lqcpwgcg@mail.gmail.com> <CAAK044SJWsvqWf+Tt3wUeGpMRH6aVg175CFUBrsz_YyhDsKYwQ@mail.gmail.com> <CADVnQy=0yhx9U-ogVX_Dh66fZWGyMzqtWAgSfaYXX-6ppGx=Kg@mail.gmail.com> <CAAK044So4qO4zma-qdYbMrJcNVdRi1-o30wcP7QjKLuT2c_Zuw@mail.gmail.com> <CADVnQynvFY24cNfe4tfm6FMY47UaMPjxrtD5RwWhCPK6EH4=6w@mail.gmail.com> <CAAK044QWL2xBvg4S_cvHFE_iTddSmnEOkhu33pftvUiUCCGZ_g@mail.gmail.com> <44e1fff-f915-fdd-6c8e-4cd1cc3a9df3@cs.helsinki.fi> <CAAK044TT93bBTkF_J8HgXxLNGnp+LLZvySf+4TOdsc=_yD8HKQ@mail.gmail.com> <9e3efc1f-e6e-8edb-d354-f442d78967ef@cs.helsinki.fi> <CAAK044SE46f6RaqcRgFOhPtOJcY77OZGEMCPEGoQMCy+2E51Ww@mail.gmail.com> <CADVnQymUztV8qV3SLeuWV1LxJRkO1SRQxxT_erqPa7owiMSCPw@mail.gmail.com> <CA+s1KQFVDbCiZDqjVfYG-feRztkUDjymV2fBy+-nGkGL8Zx9WA@mail.gmail.com> <CADVnQynAOS=ugQ6KWsORi1xqYNShH8jCp2fuVEP2ij3YvexFGQ@mail.gmail.com> <CA+s1KQEape0k1GJ+0ZRQT5BvsVd80N8F8kTJcKzGnaRods5VMg@mail.gmail.com> <EDD5465A-FE29-47D2-A1B7-5522111B9C93@ifi.uio.no> <CADVnQy=xvBmjVu6zgxuuW3QPFnxG7HgcOfxhEeJ+2PE_EyR8EQ@mail.gmail.com> <44A27685-98B7-4ACC-8B43-FE179AC0A805@ifi.uio.no>
In-Reply-To: <44A27685-98B7-4ACC-8B43-FE179AC0A805@ifi.uio.no>
From: Neal Cardwell <ncardwell@google.com>
Date: Sat, 20 Jul 2024 12:05:16 -0400
Message-ID: <CADVnQykGSYWQs_fn9JpAuzRuC+PNLQd4vv1pmNYGoGa2f+wQJw@mail.gmail.com>
To: Michael Welzl <michawe@ifi.uio.no>
Content-Type: multipart/alternative; boundary="0000000000007da58c061daffb9d"
Message-ID-Hash: UVJ2HXO257QKSVKPLUFTHCQBNX4TPRIZ
X-Message-ID-Hash: UVJ2HXO257QKSVKPLUFTHCQBNX4TPRIZ
X-MailFrom: ncardwell@google.com
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-tcpm.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
CC: Matt Mathis <matt.mathis@gmail.com>, Markku Kojo <kojo@cs.helsinki.fi>, Matt Mathis <ietf@mattmathis.net>, Matt Mathis <mattmathis@measurementlab.net>, tcpm@ietf.org
X-Mailman-Version: 3.3.9rc4
Precedence: list
Subject: [tcpm] Re: About growing cwnd when the sender is rate-limited
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/twGaavgMkvhBvUbt_GjUOrlvuQg>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Owner: <mailto:tcpm-owner@ietf.org>
List-Post: <mailto:tcpm@ietf.org>
List-Subscribe: <mailto:tcpm-join@ietf.org>
List-Unsubscribe: <mailto:tcpm-leave@ietf.org>

On Sat, Jul 20, 2024 at 9:34 AM Michael Welzl <michawe@ifi.uio.no> wrote:

>
>
> On Jul 20, 2024, at 6:20 AM, Neal Cardwell <ncardwell@google.com> wrote:
>
>
>
> On Fri, Jul 19, 2024 at 8:50 PM Michael Welzl <michawe@ifi.uio.no> wrote:
>
>> Hi,
>>
>> I changed the subject to avoid causing a distraction in the PRR thread -
>> but I couldn’t resist pointing to our draft in response to this:
>>
>>
>> On Jul 18, 2024, at 9:32 AM, Matt Mathis <matt.mathis@gmail.com> wrote:
>>
>>>
>>> On Thu, Jul 18, 2024 at 10:18 AM Matt Mathis <matt.mathis@gmail.com>
>>> wrote:
>>>
>>>> In the presentation, I would go even stronger than the text in the
>>>> document:
>>>>
>>>> (..)
>>>>
>>>
>> There are other gaping holes in the standards that are much more serious:
>>>> Not  forbidding raising cwnd when it is not being used, and
>>>>
>>>
>> (..)
>>
>> Please see draft-welzl-ccwg-ratelimited-increase-02, as this is our
>> attempt to address exactly that.
>>
>> (btw, just “forbidding” it is too strict: it should be allowed to do a
>> CC. increase based on what was really used (i.e., FlightSize) in the
>> previous RTT. We came up with a way to do this by keeping track of the max.
>> FS since the last time that cwnd was decreased, and then it turned out that
>> whoever wrote the code for Linux Slow Start had the exact same idea  :-)   )
>>
>
> Yes, that was my  commit from 2014 that changed the Linux TCP logic to
> consider what was really sent (i.e., FlightSize) in the previous RTT  when
> deciding how far cwnd can be raised (in slow-start):
>
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ca8a22634381537c92b5a10308652e1c38fd9edf
> commit ca8a22634381537c92b5a10308652e1c38fd9edf
> Author: Neal Cardwell <ncardwell@google.com>
> Date:   Thu May 22 10:41:08 2014 -0400
>     tcp: make cwnd-limited checks measurement-based, and gentler
>
> The rationale and context is described a bit in the commit message and
> comments.
>
> best regards,
> neal
>
>
> Great!  Thanks for the pointer to the commit!
>
> BTW, while this is more important in Slow Start, I think it’s still a
> mistake that (as far as I can see) the Linux code doesn’t do this in
> Congestion Avoidance. Why not do it there too?
> For this, one just needs to reset the maximum whenever cwnd is decreased
> (this is part of our definition of “maxFS” in the draft).
>

Good question, and worth a discussion.

Let me try to sketch my current thinking around the different treatment of
slow-start (is cwnd < 2 * max_packets_out?) and congestion avoidance (is
cwnd fully utilized?) in
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ca8a22634381537c92b5a10308652e1c38fd9edf
 :

Rationale 1: the likely safe degree of cwnd growth
----------------

+ In slow start, it is most often the case that doubling the cwnd *is* safe
(will not cause packet loss). That's because with exponential doubling of
cwnd each round trip, most of the time is spent with cwnd below half the
value that will cause loss. (Only in the last round trip before loss is the
cwnd within a factor of 2 of the value that will cause loss.)

+ By contrast, in congestion avoidance in steady-state, it is most often
the case that doubling the cwnd is *not* safe (*will* cause packet loss).
That's because in Reno or CUBIC congestion avoidance, given their cwnd
reduction to 0.5x or 0.7x upon loss, most of the time is spent with a cwnd
that is roughly 0.75x or more of the cwnd that last caused loss, so that
most of the time doubling the cwnd would produce a value that is >= 2 *
0.75 * W_max = 1.5 * W_max and thus would cause massive packet loss (the
expected packet loss rate is >= 50%).

Rationale 2: cost/benefit trade-offs
----------------

+ In slow start it is likely that cwnd is far below the value needed to
fully achieve a flow's fair share of the bandwidth, so there is a large
benefit to allowing doubling of the cwnd. This large benefit can balance
out a potentially significant risk of high loss (high cost) from doubling
the cwnd.

+ In slow start with a relatively stable number of flows sharing a
bottleneck, it is likely that cwnd is close to the value needed to fully
achieve a flow's fair share of the bandwidth. So there is a small benefit
to allowing doubling of the cwnd, which does not match the significant risk
of high loss (high cost) from doubling the cwnd.

Rationale 3: justifying acceleration in the convex region of CA
----------------

With the most widely deployed CC on the Internet, CUBIC, part of the
rationale for the accelerating growth of cwnd in the convex region of
congestion avoidance is: with cwnd above W_max, the longer a flow has
proceeded in accelerating its sending rate without experiencing loss, the
more confident we are that it is safe to grow cwnd with increasingly large
increments. But AFAICT that rationale only makes sense if the cwnd is fully
utilized, i.e., if the sender is actually accelerating.

For example, if we imagine a cwnd growing in a convex fashion
(qualitatively similar to CUBIC) above W_max = 1,000 MSS, and if we imagine
we use an approach like draft-welzl-ccwg-ratelimited-increase-02, and don't
check whether cwnd is fully utilized, but allow cwnd to grow to 2x the max
FlightSize, then we get an implict rationale of the flavor:

[round #] : [FlightSize sent, CC decision for cwnd]
-------------------------------------------------------
 1: FlightSize = 1,000 MSS worked, cwnd = 1,001 is fine
 2: FlightSize = 1,000 MSS worked, cwnd = 1,002 is fine
 3: FlightSize = 1,000 MSS worked, cwnd = 1,005 is fine
 4: FlightSize = 1,000 MSS worked, cwnd = 1,010 is fine
 5: FlightSize = 1,000 MSS worked, cwnd = 1,020 is fine
 6: FlightSize = 1,000 MSS worked, cwnd = 1,050 is fine
 7: FlightSize = 1,000 MSS worked, cwnd = 1,125 is fine
 8: FlightSize = 1,000 MSS worked, cwnd = 1,250 is fine
 9: FlightSize = 1,000 MSS worked, cwnd = 1,500 is fine
10: FlightSize = 1,000 MSS worked, cwnd = 2,000 is fine

(CUBIC grows more slowly than that, but this is the qualitative behavior.
And AFAICT the argument holds for any CC with some phase of convex cwnd
growth. BBRv2 / BBRv3 have qualitatively similar behavior in this regard.)

IMHO that behavior does not make sense, or at least seems needlessly risky:

(a) Why would a congestion control algorithm deem that it can safely
accelerate its growth of cwnd above W_max = 1,000 in this way if FlightSize
never gets above 1,000 packets?

(b) What is the expected level of loss in this scenario if the flow
transitions back to fully utilizing its cwnd? If the "safe" (no-loss)
FlightSize really is still 1,000 MSS, then the expected loss rate in this
case when fully using this cwnd and sending 2,000 MSS is a 50% packet loss
rate (1,000 MSS lost out of 2,000 MSS sent).

----------------

That said, I agree that it's worth further discussion of, and possibly
experiments with, approaches more like
draft-welzl-ccwg-ratelimited-increase-02, allowing cwnd to grow to 2 *
max_packets_out in any state. Partly because it's easy to come up with
theoretical arguments (like mine above) for why a CC approach is too
aggressive, and sometimes aggressive approaches work surprisingly well in
practice (I am thinking here of slow start...).

cheers,
neal