Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set cwnd to ssthresh exiting fast recovery?
Yoshifumi Nishida <nsd.ietf@gmail.com> Wed, 09 August 2023 07:16 UTC
Return-Path: <nsd.ietf@gmail.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A259CC15107B for <tcpm@ietfa.amsl.com>; Wed, 9 Aug 2023 00:16:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.005
X-Spam-Level:
X-Spam-Status: No, score=-2.005 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0Ot0ICQXcySa for <tcpm@ietfa.amsl.com>; Wed, 9 Aug 2023 00:16:27 -0700 (PDT)
Received: from mail-ot1-x336.google.com (mail-ot1-x336.google.com [IPv6:2607:f8b0:4864:20::336]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 99CFFC151096 for <tcpm@ietf.org>; Wed, 9 Aug 2023 00:16:27 -0700 (PDT)
Received: by mail-ot1-x336.google.com with SMTP id 46e09a7af769-6bd0afbd616so1023381a34.0 for <tcpm@ietf.org>; Wed, 09 Aug 2023 00:16:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691565387; x=1692170187; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=JYaxX9jmuE7xUACpF41FaMliuGJDhMcURsuYXpQ52KI=; b=PJEiLBzl9rpM/ek6HCg8USvYjnTqzbJfBwWkcHKBISdASoybls8kfxPByA+Pzi0B7b NMORoOotNUw042OsSoQxA0pnhAZ0kNeKcKJUDwCfTPxlLK65lxWyelH5c5qpJlpOiW7Q pdDFqKjAEKjjLJU4cT+rblU8bUNu491T0DM85Tq7cBtmoRSkLIRKc1q35a3TzQRdgr5i h+W39jMUm1OimHq1z7aMgBubJELOC2BiMVYSb04CO/X0n1n67/QrWkV6Uh/DKAjyganq MsfNToh1mhE+KG/G1nxs5a6v9Rh6vc3Rf3h69vANisc8mXgl0w4w+y5h0jCtlm9+5e3d 7yHQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691565387; x=1692170187; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JYaxX9jmuE7xUACpF41FaMliuGJDhMcURsuYXpQ52KI=; b=baEmF/aP4M9izSz6rzTUn6czmmBcql4/GMZPf2Sw876CofWFF6PYZV/jMWoSUsqlKr u/oKvmCC20rKEuxT9rKUrE/TX1+U1wsTu9RcN1ecFWKqj7jqc0mDIpSUc4WUZN/Do6X/ 1OsuiLfHdPkk/m9wVX7IWSsSiA+8hhWB4YjX6aTTqj2ilkGexccijADLTZn69PiJYPJ+ OX9ulF4eOW8vwvjUoapeDxSz1MuJHG/t94PiCs9ZasldkrLQAEV/VhVTPrMZy31U5oN8 F1CV+bRsrtMJvEOTddVneZSTeC/TyfAORIDy7OMa6nHXKeM5S1avTCg26/UEogzMCjkM gbrA==
X-Gm-Message-State: AOJu0YxphTXC0HRJ/oVRMNDkdvO9wPvTys1j0ZEOVMKI3S8/utxJCnNN npmDpXBJ0Rlvw5q87G72xQ7oDdwQ5+9aFLEy5Oux7RSI
X-Google-Smtp-Source: AGHT+IFZbxZ47YZIuc2E6CMRq7f6h6FSyZbv+FGpou8/Nkb3kLjnhrKqqoWQ3a+NmAe1vUsM03nG4U/hPXftNH10/EI=
X-Received: by 2002:a05:6870:f61d:b0:1bb:9de5:badb with SMTP id ek29-20020a056870f61d00b001bb9de5badbmr2266915oab.19.1691565386620; Wed, 09 Aug 2023 00:16:26 -0700 (PDT)
MIME-Version: 1.0
References: <CADVnQy=rbTc1rb5PKA1mvSJm61UTb=T5xzOkMBBB2Yadoe691A@mail.gmail.com> <CAK6E8=ckFHoiRTmLEy6ZH8z2ovv9+7S_UzUqnO3W4xcumyA1Gg@mail.gmail.com> <CADVnQyk7nxmaoTHh5qo9XvhrWojoB2R78FK0zX5CcwoZq6c=hg@mail.gmail.com> <CAK6E8=cXXWfHd+T3GkDEhJ6TmbstygL=qD4nns3w50DTe2eaZw@mail.gmail.com> <CADVnQy=Q5cvN_+Fa0rbNc2a_Aqe=haROOd4SNpk9TbvE1MXVvQ@mail.gmail.com> <CADVnQymCZkqRw6f8JTuFXhNXEo1KJx4S48gXaBaOPRasOVCg+Q@mail.gmail.com> <CAAK044QCh_KyFugteUo1eaez_6LipCXtJKW1rxaHqhidfRRGmQ@mail.gmail.com> <F24D815E-4932-4A84-B6C6-ECBCEB487199@netflix.com> <CAAK044QvbVHs+eFfitxpDUQOM2_vtBei-p5+ZUcatXTyYYE++g@mail.gmail.com> <CADVnQyn-Oi+0XpZMa9KLPdSMwCYpB-PQNYb0f6xRB6FeCMteoA@mail.gmail.com> <CAAK044RR1Vd3tNhsUXH4Ce66BVwg_z+O-vOrACmiOzf-+avS8A@mail.gmail.com> <CAAK044QDjUej5Z=Q32i+P6zJe72ZnSDF0JJjkqrEN5zSHtqwYQ@mail.gmail.com> <CAK6E8=depqAzh0FN0JYkXOWYsZU-bGfXybaqj4Jn2sySKb9_rg@mail.gmail.com>
In-Reply-To: <CAK6E8=depqAzh0FN0JYkXOWYsZU-bGfXybaqj4Jn2sySKb9_rg@mail.gmail.com>
From: Yoshifumi Nishida <nsd.ietf@gmail.com>
Date: Wed, 09 Aug 2023 00:16:15 -0700
Message-ID: <CAAK044T6GX6mC0oX=f46PjJScx5ah4hivvhYAcZ_TbBUj1nMtw@mail.gmail.com>
To: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>, tcpm <tcpm@ietf.org>
Content-Type: multipart/alternative; boundary="000000000000fbe4a50602784116"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/kR0Vdid_bmZH8IHmPcQVyChWBVk>
Subject: Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set cwnd to ssthresh exiting fast recovery?
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Aug 2023 07:16:28 -0000
Hi Yuchung, Thanks for the response. I just would like to check one thing. In my understanding, Neal's suggestion here was to adjust cwnd to ssthresh at the end of recovery. But, I cannot find the statement or logic for such adjustment. Does this mean we decided there's no adjustment at the end of recovery? Or, am I missing something? Thanks, -- Yoshi On Tue, Aug 8, 2023 at 2:34 PM Yuchung Cheng <ycheng@google.com> wrote: > Hi Yoshifumi, > > That part is how the "RecoverFS" state variable is calculated in the > draft. See the diff of 03/04 on Section 5 and 6 regarding "RecoverFS" state > variable definition and computation. > https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-prr-rfc6937bis-04 > > Does that make sense? > > On Tue, Aug 8, 2023 at 12:01 AM Yoshifumi Nishida <nsd.ietf@gmail.com> > wrote: > >> Hi Yuchung, >> >> I think you have already updated the draft on the following point from >> the discussions in the last WG meeting. >> Could you point out which part has been updated? I'm just checking.. >> Thanks, >> -- >> Yoshi >> >> On Fri, May 5, 2023 at 11:51 AM Yoshifumi Nishida <nsd.ietf@gmail.com> >> wrote: >> >>> Hi Neal, >>> >>> Yes, I think I understand your point. >>> I prefer the current logic in some ways as it's more conservative as I >>> think we cannot always presume that queue has been drained at the end of >>> recovery. >>> But, I also think it may look too conservative. >>> I am expecting that the authors provide some insights on this point. >>> -- >>> Yoshi >>> >>> >>> On Tue, May 2, 2023 at 11:31 AM Neal Cardwell <ncardwell@google.com> >>> wrote: >>> >>>> Hi Yoshi, >>>> >>>> You are right that because PRR always sets cwnd to ssthresh at the end >>>> of recovery, there will be some cases where with PRR cwnd jumps up >>>> drastically at the end of the recovery. >>>> >>>> However, AFAIK cwnd jumping up drastically, per se, is not a problem. >>>> Big bursts of packets going into the network is a problem. And given the >>>> dynamics of the alternative loss recovery algorithms (RFC6675 and PRR), >>>> both can allow bursts of packets; just in different circumstances: >>>> >>>> (1) RFC6675: Because RFC6675 sets cwnd once at the start of fast >>>> recovery, using (4.2) from RFC6675: >>>> >>>> ssthresh = cwnd = (FlightSize / 2) >>>> >>>> ...that means RFC6675 allows big bursts at the moment any loss is >>>> detected: any time L packets are lost, the sender can burst L more packets. >>>> >>>> (2) PRR: PRR is specifically designed to avoid big bursts in response >>>> to packet losses; no matter the structure or timing of the losses, PRR only >>>> allows a big burst at the end of Fast Recovery after all holes have been >>>> plugged, and the algorithm sets cwnd to ssthresh. >>>> >>>> So in your example ("For example, many packets were lost before >>>> entering recovery"), AFAICT RFC6675 can allow a big burst at the beginning >>>> of recovery, when the lost packets are detected. AFAICT in this case PRR >>>> can allow a burst of packets at the end of recovery when it sets cwnd to >>>> ssthresh, but at least at this point the bottleneck queue has potentially >>>> drained somewhat. >>>> >>>> Please let me know if that analysis misses something important. :-) >>>> >>>> Thanks! >>>> neal >>>> >>>> >>>> On Mon, May 1, 2023 at 5:22 PM Yoshifumi Nishida <nsd.ietf@gmail.com> >>>> wrote: >>>> >>>>> Hi Randall, >>>>> >>>>> I might miss something, but here's what I've thought.. >>>>> If we lost many packets in a RTT such as the Figure 5 in the 6937bis >>>>> draft, I think the window growth during the recovery period will be bound >>>>> by PRR-CRB or PRR-SSRB. >>>>> Hence, I think the cwnd at the end of recovery can be smaller than we >>>>> expect as shown in figure 5. >>>>> -- >>>>> Yoshi >>>>> >>>>> On Mon, May 1, 2023 at 4:17 AM Randall Stewart <rrs@netflix.com> >>>>> wrote: >>>>> >>>>>> Hi Neal and Yoshi: >>>>>> >>>>>> Neal: So the FreeBSD implementation in rack, like linux, does the >>>>>> same exact thing set cwnd to ssthresh at >>>>>> exit from recovery. >>>>>> >>>>>> Yoshi: I don’t see how this would cause cwnd to be larger, since at >>>>>> the entry to recovery you set ssthresh = cwnd * Beta. But >>>>>> maybe I am missing something, can you give an example like >>>>>> Neal did below? >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> R >>>>>> >>>>>> On May 1, 2023, at 5:32 AM, Yoshifumi Nishida <nsd.ietf@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Hi Neal, >>>>>> >>>>>> If we always set cwnd to ssthresh at the end of recovery, I am >>>>>> guessing there will be some cases where cwnd jumps up drastically at the >>>>>> end of the recovery. For example, many packets were lost before entering >>>>>> recovery. Or, am I missing something? >>>>>> -- >>>>>> Yoshi >>>>>> >>>>>> On Wed, Apr 19, 2023 at 7:37 PM Neal Cardwell <ncardwell= >>>>>> 40google.com@dmarc.ietf.org> wrote: >>>>>> >>>>>>> Working through examples for the "draft-ietf-tcpm-prr-rfc6937bis-03 >>>>>>> and RecoverFS initialization" thread this evening, I ran into another >>>>>>> potential issue. >>>>>>> >>>>>>> The Linux TCP implementation of PRR explicitly/directly sets cwnd to >>>>>>> ssthresh at the end of fast recovery (in tcp_end_cwnd_reduction()). But >>>>>>> this behavior is not in the algorithm in the PRR RFC or draft, at least in >>>>>>> the figures in section 6, Algorithms. Maybe it is in the prose somewhere >>>>>>> and I missed it; but in that case I'd argue strongly to put this in the >>>>>>> figures in section 6, Algorithms. >>>>>>> >>>>>>> AFAICT in some cases this is strictly necessary to get cwnd to grow >>>>>>> to reach ssthresh. Without such a direct step, cwnd could end up far below >>>>>>> ssthresh at the end of recovery. Here's an example to illustrate: >>>>>>> >>>>>>> CC = CUBIC >>>>>>> >>>>>>> cwnd = 10 >>>>>>> >>>>>>> The reordering degree was estimated to be large, so the connection >>>>>>> will wait for more than 3 packets to be SACKed before entering fast >>>>>>> recovery. >>>>>>> >>>>>>> --- Application writes 10*MSS. >>>>>>> >>>>>>> TCP sends packets P1 .. P10. >>>>>>> pipe = 10 packets in flight (P1 .. P10) >>>>>>> >>>>>>> --- P2..P9 SACKed -> do nothing >>>>>>> >>>>>>> (Because the reordering degree was previously estimated to be large.) >>>>>>> >>>>>>> --- P10 SACKed -> mark P1 as lost and enter fast recovery >>>>>>> >>>>>>> PRR: >>>>>>> ssthresh = CongCtrlAlg() = 7 packets // CUBIC >>>>>>> prr_delivered = 0 >>>>>>> prr_out = 0 >>>>>>> RecoverFS = snd.nxt - snd.una = 10 packets (P1..P10) >>>>>>> >>>>>>> DeliveredData = 1 (P10 was SACKed) >>>>>>> >>>>>>> prr_delivered += DeliveredData ==> prr_delivered = 1 >>>>>>> >>>>>>> pipe = 0 (all packets are SACKed or lost; P1 is lost, rest are >>>>>>> SACKed) >>>>>>> >>>>>>> safeACK = false (snd.una did not advance) >>>>>>> >>>>>>> if (pipe > ssthresh) => if (0 > 7) => false >>>>>>> else >>>>>>> // PRR-CRB by default >>>>>>> sndcnt = MAX(prr_delivered - prr_out, DeliveredData) >>>>>>> = MAX(1 - 0, 1) >>>>>>> = 1 >>>>>>> >>>>>>> sndcnt = MIN(ssthresh - pipe, sndcnt) >>>>>>> = MIN(7 - 0, 1) >>>>>>> = 1 >>>>>>> >>>>>>> cwnd = pipe + sndcnt >>>>>>> = 0 + 1 >>>>>>> = 1 >>>>>>> >>>>>>> retransmit P1 >>>>>>> >>>>>>> prr_out += 1 ==> prr_out = 1 >>>>>>> >>>>>>> --- P1 retransmit plugs hole; receive cumulative ACK for P1..P10 >>>>>>> >>>>>>> DeliveredData = 1 (P1 was newly ACKed) >>>>>>> >>>>>>> prr_delivered += DeliveredData ==> prr_delivered = 2 >>>>>>> >>>>>>> pipe = 0 (all packets are cumuatively ACKed) >>>>>>> >>>>>>> safeACK = (snd.una advances and no further loss indicated) >>>>>>> safeACK = true >>>>>>> >>>>>>> if (pipe > ssthresh) => if (0 > 7) => false >>>>>>> else >>>>>>> // PRR-CRB by default >>>>>>> sndcnt = MAX(prr_delivered - prr_out, DeliveredData) >>>>>>> = MAX(2 - 1, 1) >>>>>>> = 1 >>>>>>> if (safeACK) => true >>>>>>> // PRR-SSRB when recovery is in good progress >>>>>>> sndcnt += 1 ==> sndcnt = 2 >>>>>>> >>>>>>> sndcnt = MIN(ssthresh - pipe, sndcnt) >>>>>>> = MIN(7 - 0, 2) >>>>>>> = 2 >>>>>>> >>>>>>> cwnd = pipe + sndcnt >>>>>>> = 0 + 2 >>>>>>> = 2 >>>>>>> >>>>>>> So we exit fast recovery with cwnd=2 even though ssthresh is 7. >>>>>>> >>>>>>> As noted above, the Linux TCP implementation does not suffer this >>>>>>> problem because it explicitly/directly sets cwnd to ssthresh at the end of >>>>>>> fast recovery. >>>>>>> >>>>>>> I would recommend including this cwnd=ssthresh step at the end of >>>>>>> recovery in the draft, to ensure that cwnd reaches ssthresh at the end of >>>>>>> fast recovery, even in cases like this where there will be insufficient >>>>>>> delivered data in fast recovery to allow pipe to incrementally grow to >>>>>>> reach ssthresh using PRR-SSRB. >>>>>>> >>>>>>> Best regards, >>>>>>> neal >>>>>>> >>>>>>> _______________________________________________ >>>>>>> tcpm mailing list >>>>>>> tcpm@ietf.org >>>>>>> https://www.ietf.org/mailman/listinfo/tcpm >>>>>>> <https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw> >>>>>>> >>>>>> _______________________________________________ >>>>>> tcpm mailing list >>>>>> tcpm@ietf.org >>>>>> >>>>>> https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw >>>>>> >>>>>> >>>>>> ------ >>>>>> Randall Stewart >>>>>> rrs@netflix.com >>>>>> >>>>>> >>>>>> >>>>>>
- [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and Reco… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Yuchung Cheng
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Yuchung Cheng
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Neal Cardwell
- [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set cwn… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Randall Stewart
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yuchung Cheng
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yuchung Cheng
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Yoshifumi Nishida
- Re: [tcpm] [EXTERNAL] Re: draft-ietf-tcpm-prr-rfc… Yi Huang
- Re: [tcpm] [EXTERNAL] Re: draft-ietf-tcpm-prr-rfc… Yuchung Cheng
- Re: [tcpm] [EXTERNAL] Re: draft-ietf-tcpm-prr-rfc… Yi Huang
- Re: [tcpm] [EXTERNAL] Re: draft-ietf-tcpm-prr-rfc… Yuchung Cheng
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Neal Cardwell
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set… Randall Stewart
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Yoshifumi Nishida
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Yuchung Cheng
- Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03 and … Yoshifumi Nishida