Re: [tcpm] [EXTERNAL] Re: draft-ietf-tcpm-prr-rfc6937bis-03: set cwnd to ssthresh exiting fast recovery?

[Yuchung Cheng <[e-mail]>]

Thank you Yi. I assume Windows TCP simply transmit the amount "sndcnt"
indicates, w/o changing cwnd, correct?

(In that case it's fine, and we can mention that in the draft for
compatibility reasons)

On Mon, Aug 21, 2023 at 11:09 AM Yi Huang <[e-mail]> wrote:

> Sorry, I didn't know there were questions for Windows TCP. Just providing
> a data point: we set cwnd to ssthresh upon entering loss recovery. Also,
> our PRR implementation does not modify cwnd during loss recovery (which I
> believe is consistent with the original RFC6937).
> ------------------------------
> *From:* Yoshifumi Nishida <[e-mail]>
> *Sent:* Monday, August 21, 2023 10:41 AM
> *To:* Neal Cardwell <[e-mail]>
> *Cc:* Yuchung Cheng <[e-mail]>; tcpm <[e-mail]>; Matt Olson
> <[e-mail]>; Yi Huang <[e-mail]>
> *Subject:* [EXTERNAL] Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set
> cwnd to ssthresh exiting fast recovery?
>
> Hello everyone,
>
> If there are no other particular options on this, I think the authors
> would update the draft based on the discussions or send proposed texts to
> the ML.
> Either way, after reviewing the updated texts, the chairs will think about
> whether we can conclude the WGLC for the draft.
> If you have any opinions, please let us know.
>
> Thanks!
> --
> Yoshi
>
> On Wed, Aug 9, 2023 at 11:27 AM Yoshifumi Nishida <[e-mail]>
> wrote:
>
> Hi Neal, Yuchung,
>
> Thank you so much.
> This sounds like a good direction. I also think using SHOULD is a good
> balance.
> If other people especially working on implementations have some thoughts,
> please share.
> --
> Yoshi
>
> On Wed, Aug 9, 2023 at 8:45 AM Neal Cardwell <[e-mail]> wrote:
>
> On Wed, Aug 9, 2023 at 11:39 AM Yuchung Cheng <[e-mail]> wrote:
>
> We can add "Upon exiting recovery, cwnd /SHOULD/ be set to ssthresh" with
> some performance rationale, given that RFC6675 and existing PRR
> implementation already do so.
>
>
> This sounds very good to me. Yoshifumi, how does that sound?
>
> thanks,
> neal
>
>
> Note that RFC5681 Sec 4.3 has related wording on cwnd exiting recovery:
> "Finally, after all loss in the given window of segments has been
> successfully retransmitted, cwnd MUST be set to no more than ssthresh and
> congestion avoidance MUST be used to further increase cwnd."
>
> Why not MUST: it's not strictly necessary because it won't break TCP or
> make network unstable. It's important for congestion control to determine
> the cwnd after recovery. It has the implication to induce a large burst if
> cwnd >> pipe as mentioned in the end of Section 5 in RFC6675.
> Why not MAY: lacking so has major performance implications in various
> cases as discussed in this thread
>
> How does that sound?
>
> On Wed, Aug 9, 2023 at 8:10 AM Neal Cardwell <[e-mail]> wrote:
>
>
>
> On Wed, Aug 9, 2023 at 11:05 AM Neal Cardwell <[e-mail]>
> wrote:
>
> Hi Yoshifumi,
>
> You are correct that draft-ietf-tcpm-prr-rfc6937bis-04 does not
>  incorporate the suggestion in this thread to have a "cwnd = ssthresh" step
> at the end of fast recovery. My sense was that this was because we had not
> come to a conclusion / resolution of this question in this thread. :-)
>
> I would still argue that it's important for PRR to set cwnd = ssthresh at
> the end of recovery. Without setting cwnd = ssthresh at the end of
> recovery, cwnd could end recovery far below ssthresh, leading to unusably
> terrible performance; performance that would be far worse than RFC 6675
> recovery (which simply sets cwnd = ssthresh at the start of recovery).
>
> The Linux TCP PRR has had this cwnd = ssthresh step at the end of recovery
> since the original PRR implementation in 2011:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a262f0cdf1f2916ea918dc329492abb5323d9a6c
>
>
> And FWIW it sounds like from Randall Stewart's earlier post on this thread
> ("when we exit recovery we set cwnd to ssthresh") that FreeBSD TCP PRR also
> has the same  cwnd = ssthresh step at the end of recovery that Linux TCP
> PRR has.
>
> I would suspect that Microsoft TCP PRR has a similar step; I've CC-ed some
> folks who may be able to shed light on that.
>
> neal
>
>
>
> best regards,
> neal
>
>
>
>
> On Wed, Aug 9, 2023 at 3:16 AM Yoshifumi Nishida <[e-mail]>
> wrote:
>
> Hi Yuchung,
>
> Thanks for the response.
> I just would like to check one thing.
> In my understanding, Neal's suggestion here was to adjust cwnd to ssthresh
> at the end of recovery.
> But, I cannot find the statement or logic for such adjustment. Does this
> mean we decided there's no adjustment at the end of recovery? Or, am I
> missing something?
>
> Thanks,
> --
> Yoshi
>
> On Tue, Aug 8, 2023 at 2:34 PM Yuchung Cheng <[e-mail]> wrote:
>
> Hi Yoshifumi,
>
> That part is how the "RecoverFS" state variable is calculated in the
> draft. See the diff of 03/04 on Section 5 and 6 regarding "RecoverFS" state
> variable definition and computation.
> https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-prr-rfc6937bis-04
>
> Does that make sense?
>
> On Tue, Aug 8, 2023 at 12:01 AM Yoshifumi Nishida <[e-mail]>
> wrote:
>
> Hi Yuchung,
>
> I think you have already updated the draft on the following point from the
> discussions in the last WG meeting.
> Could you point out which part has been updated? I'm just checking..
> Thanks,
> --
> Yoshi
>
> On Fri, May 5, 2023 at 11:51 AM Yoshifumi Nishida <[e-mail]>
> wrote:
>
> Hi Neal,
>
> Yes, I think I understand your point.
> I prefer the current logic in some ways as it's more conservative as I
> think we cannot always presume that queue has been drained at the end of
> recovery.
> But, I also think it may look too conservative.
> I am expecting that the authors provide some insights on this point.
> --
> Yoshi
>
>
> On Tue, May 2, 2023 at 11:31 AM Neal Cardwell <[e-mail]>
> wrote:
>
> Hi Yoshi,
>
> You are right that because PRR always sets cwnd to ssthresh at the end of
> recovery, there will be some cases where with PRR cwnd jumps up drastically
> at the end of the recovery.
>
> However, AFAIK cwnd jumping up drastically, per se, is not a problem. Big
> bursts of packets going into the network is a problem. And given the
> dynamics of the alternative loss recovery algorithms (RFC6675 and PRR),
> both can allow bursts of packets; just in different circumstances:
>
> (1) RFC6675: Because RFC6675 sets cwnd once at the start of fast recovery,
> using (4.2) from RFC6675:
>
> ssthresh = cwnd = (FlightSize / 2)
>
> ...that means RFC6675 allows big bursts at the moment any loss is
> detected: any time L packets are lost, the sender can burst L more packets.
>
> (2) PRR: PRR is specifically designed to avoid big bursts in response to
> packet losses; no matter the structure or timing of the losses, PRR only
> allows a big burst at the end of Fast Recovery after all holes have been
> plugged, and the algorithm sets cwnd to ssthresh.
>
> So in your example ("For example, many packets were lost before entering
> recovery"), AFAICT RFC6675 can allow a big burst at the beginning of
> recovery, when the lost packets are detected. AFAICT in this case PRR can
> allow a burst of packets at the end of recovery when it sets cwnd to
> ssthresh, but at least at this point the bottleneck queue has potentially
> drained somewhat.
>
> Please let me know if that analysis misses something important. :-)
>
> Thanks!
> neal
>
>
> On Mon, May 1, 2023 at 5:22 PM Yoshifumi Nishida <[e-mail]>
> wrote:
>
> Hi Randall,
>
> I might miss something, but here's what I've thought..
> If we lost many packets in a RTT such as the Figure 5 in the 6937bis
> draft, I think the window growth during the recovery period will be bound
> by PRR-CRB or PRR-SSRB.
> Hence, I think the cwnd at the end of recovery can be smaller than we
> expect as shown in figure 5.
> --
> Yoshi
>
> On Mon, May 1, 2023 at 4:17 AM Randall Stewart <[e-mail]> wrote:
>
> Hi Neal and Yoshi:
>
> Neal: So the FreeBSD implementation in rack, like linux, does the same
> exact thing set cwnd to ssthresh at
> exit from recovery.
>
> Yoshi: I don’t see how this would cause cwnd to be larger, since at the
> entry to recovery you set ssthresh = cwnd *  Beta. But
>           maybe I am missing something, can you give an example like Neal
> did below?
>
>
> Thanks
>
> R
>
> On May 1, 2023, at 5:32 AM, Yoshifumi Nishida <[e-mail]> wrote:
>
> Hi Neal,
>
> If we always set cwnd to ssthresh at the end of recovery, I am guessing
> there will be some cases where cwnd jumps up drastically at the end of the
> recovery. For example, many packets were lost before entering recovery.
> Or, am I missing something?
> --
> Yoshi
>
> On Wed, Apr 19, 2023 at 7:37 PM Neal Cardwell <ncardwell=
> [e-mail]> wrote:
>
> Working through examples for the "draft-ietf-tcpm-prr-rfc6937bis-03 and
> RecoverFS initialization" thread this evening, I ran into another potential
> issue.
>
> The Linux TCP implementation of PRR explicitly/directly sets cwnd to
> ssthresh at the end of fast recovery (in tcp_end_cwnd_reduction()). But
> this behavior is not in the algorithm in the PRR RFC or draft, at least in
> the figures in section 6, Algorithms. Maybe it is in the prose somewhere
> and I missed it; but in that case I'd argue strongly to put this in the
> figures in section 6, Algorithms.
>
> AFAICT in some cases this is strictly necessary to get cwnd to grow to
> reach ssthresh. Without such a direct step, cwnd could end up far below
> ssthresh at the end of recovery. Here's an example to illustrate:
>
> CC = CUBIC
>
> cwnd = 10
>
> The reordering degree was estimated to be large, so the connection will
> wait for more than 3 packets to be SACKed before entering fast recovery.
>
> --- Application writes 10*MSS.
>
> TCP sends packets P1 .. P10.
> pipe = 10 packets in flight (P1 .. P10)
>
> --- P2..P9 SACKed  -> do nothing
>
> (Because the reordering degree was previously estimated to be large.)
>
> --- P10 SACKed -> mark P1 as lost and enter fast recovery
>
> PRR:
> ssthresh = CongCtrlAlg() = 7 packets // CUBIC
> prr_delivered = 0
> prr_out = 0
> RecoverFS = snd.nxt - snd.una = 10 packets (P1..P10)
>
> DeliveredData = 1  (P10 was SACKed)
>
> prr_delivered += DeliveredData   ==> prr_delivered = 1
>
> pipe =  0  (all packets are SACKed or lost; P1 is lost, rest are SACKed)
>
> safeACK = false (snd.una did not advance)
>
> if (pipe > ssthresh) => if (0 > 7) => false
> else
>   // PRR-CRB by default
>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>          = MAX(1 - 0, 1)
>          = 1
>
>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>          = MIN(7 - 0, 1)
>          = 1
>
> cwnd = pipe + sndcnt
>      = 0    + 1
>      = 1
>
> retransmit P1
>
> prr_out += 1   ==> prr_out = 1
>
> --- P1 retransmit plugs hole; receive cumulative ACK for P1..P10
>
> DeliveredData = 1  (P1 was newly ACKed)
>
> prr_delivered += DeliveredData   ==> prr_delivered = 2
>
> pipe =  0  (all packets are cumuatively ACKed)
>
> safeACK = (snd.una advances and no further loss indicated)
> safeACK = true
>
> if (pipe > ssthresh) => if (0 > 7) => false
> else
>   // PRR-CRB by default
>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>          = MAX(2 - 1, 1)
>          = 1
>   if (safeACK) => true
>     // PRR-SSRB when recovery is in good progress
>     sndcnt += 1   ==> sndcnt = 2
>
>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>          = MIN(7 - 0, 2)
>          = 2
>
> cwnd = pipe + sndcnt
>      = 0    + 2
>      = 2
>
> So we exit fast recovery with cwnd=2 even though ssthresh is 7.
>
> As noted above, the Linux TCP implementation does not suffer this problem
> because it explicitly/directly sets cwnd to ssthresh at the end of fast
> recovery.
>
> I would recommend including this cwnd=ssthresh step at the end of recovery
> in the draft, to ensure that cwnd reaches ssthresh at the end of fast
> recovery, even in cases like this where there will be insufficient
> delivered data in fast recovery to allow pipe to incrementally grow to
> reach ssthresh using PRR-SSRB.
>
> Best regards,
> neal
>
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tcpm
> <https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw>
>
> _______________________________________________
> tcpm mailing list
> [e-mail]
>
> https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw
>
>
> ------
> Randall Stewart
> [e-mail]
>
>
>
>