Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set cwnd to ssthresh exiting fast recovery?

[Neal Cardwell <[e-mail]>]

Hi Yoshifumi,

You are correct that draft-ietf-tcpm-prr-rfc6937bis-04 does not
 incorporate the suggestion in this thread to have a "cwnd = ssthresh" step
at the end of fast recovery. My sense was that this was because we had not
come to a conclusion / resolution of this question in this thread. :-)

I would still argue that it's important for PRR to set cwnd = ssthresh at
the end of recovery. Without setting cwnd = ssthresh at the end of
recovery, cwnd could end recovery far below ssthresh, leading to unusably
terrible performance; performance that would be far worse than RFC 6675
recovery (which simply sets cwnd = ssthresh at the start of recovery).

The Linux TCP PRR has had this cwnd = ssthresh step at the end of recovery
since the original PRR implementation in 2011:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a262f0cdf1f2916ea918dc329492abb5323d9a6c

best regards,
neal




On Wed, Aug 9, 2023 at 3:16 AM Yoshifumi Nishida <[e-mail]> wrote:

> Hi Yuchung,
>
> Thanks for the response.
> I just would like to check one thing.
> In my understanding, Neal's suggestion here was to adjust cwnd to ssthresh
> at the end of recovery.
> But, I cannot find the statement or logic for such adjustment. Does this
> mean we decided there's no adjustment at the end of recovery? Or, am I
> missing something?
>
> Thanks,
> --
> Yoshi
>
> On Tue, Aug 8, 2023 at 2:34 PM Yuchung Cheng <[e-mail]> wrote:
>
>> Hi Yoshifumi,
>>
>> That part is how the "RecoverFS" state variable is calculated in the
>> draft. See the diff of 03/04 on Section 5 and 6 regarding "RecoverFS" state
>> variable definition and computation.
>> https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-prr-rfc6937bis-04
>>
>> Does that make sense?
>>
>> On Tue, Aug 8, 2023 at 12:01 AM Yoshifumi Nishida <[e-mail]>
>> wrote:
>>
>>> Hi Yuchung,
>>>
>>> I think you have already updated the draft on the following point from
>>> the discussions in the last WG meeting.
>>> Could you point out which part has been updated? I'm just checking..
>>> Thanks,
>>> --
>>> Yoshi
>>>
>>> On Fri, May 5, 2023 at 11:51 AM Yoshifumi Nishida <[e-mail]>
>>> wrote:
>>>
>>>> Hi Neal,
>>>>
>>>> Yes, I think I understand your point.
>>>> I prefer the current logic in some ways as it's more conservative as I
>>>> think we cannot always presume that queue has been drained at the end of
>>>> recovery.
>>>> But, I also think it may look too conservative.
>>>> I am expecting that the authors provide some insights on this point.
>>>> --
>>>> Yoshi
>>>>
>>>>
>>>> On Tue, May 2, 2023 at 11:31 AM Neal Cardwell <[e-mail]>
>>>> wrote:
>>>>
>>>>> Hi Yoshi,
>>>>>
>>>>> You are right that because PRR always sets cwnd to ssthresh at the end
>>>>> of recovery, there will be some cases where with PRR cwnd jumps up
>>>>> drastically at the end of the recovery.
>>>>>
>>>>> However, AFAIK cwnd jumping up drastically, per se, is not a problem.
>>>>> Big bursts of packets going into the network is a problem. And given the
>>>>> dynamics of the alternative loss recovery algorithms (RFC6675 and PRR),
>>>>> both can allow bursts of packets; just in different circumstances:
>>>>>
>>>>> (1) RFC6675: Because RFC6675 sets cwnd once at the start of fast
>>>>> recovery, using (4.2) from RFC6675:
>>>>>
>>>>> ssthresh = cwnd = (FlightSize / 2)
>>>>>
>>>>> ...that means RFC6675 allows big bursts at the moment any loss is
>>>>> detected: any time L packets are lost, the sender can burst L more packets.
>>>>>
>>>>> (2) PRR: PRR is specifically designed to avoid big bursts in response
>>>>> to packet losses; no matter the structure or timing of the losses, PRR only
>>>>> allows a big burst at the end of Fast Recovery after all holes have been
>>>>> plugged, and the algorithm sets cwnd to ssthresh.
>>>>>
>>>>> So in your example ("For example, many packets were lost before
>>>>> entering recovery"), AFAICT RFC6675 can allow a big burst at the beginning
>>>>> of recovery, when the lost packets are detected. AFAICT in this case PRR
>>>>> can allow a burst of packets at the end of recovery when it sets cwnd to
>>>>> ssthresh, but at least at this point the bottleneck queue has potentially
>>>>> drained somewhat.
>>>>>
>>>>> Please let me know if that analysis misses something important. :-)
>>>>>
>>>>> Thanks!
>>>>> neal
>>>>>
>>>>>
>>>>> On Mon, May 1, 2023 at 5:22 PM Yoshifumi Nishida <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> Hi Randall,
>>>>>>
>>>>>> I might miss something, but here's what I've thought..
>>>>>> If we lost many packets in a RTT such as the Figure 5 in the 6937bis
>>>>>> draft, I think the window growth during the recovery period will be bound
>>>>>> by PRR-CRB or PRR-SSRB.
>>>>>> Hence, I think the cwnd at the end of recovery can be smaller than we
>>>>>> expect as shown in figure 5.
>>>>>> --
>>>>>> Yoshi
>>>>>>
>>>>>> On Mon, May 1, 2023 at 4:17 AM Randall Stewart <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Neal and Yoshi:
>>>>>>>
>>>>>>> Neal: So the FreeBSD implementation in rack, like linux, does the
>>>>>>> same exact thing set cwnd to ssthresh at
>>>>>>> exit from recovery.
>>>>>>>
>>>>>>> Yoshi: I don’t see how this would cause cwnd to be larger, since at
>>>>>>> the entry to recovery you set ssthresh = cwnd *  Beta. But
>>>>>>>           maybe I am missing something, can you give an example like
>>>>>>> Neal did below?
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> R
>>>>>>>
>>>>>>> On May 1, 2023, at 5:32 AM, Yoshifumi Nishida <[e-mail]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Neal,
>>>>>>>
>>>>>>> If we always set cwnd to ssthresh at the end of recovery, I am
>>>>>>> guessing there will be some cases where cwnd jumps up drastically at the
>>>>>>> end of the recovery. For example, many packets were lost before entering
>>>>>>> recovery.  Or, am I missing something?
>>>>>>> --
>>>>>>> Yoshi
>>>>>>>
>>>>>>> On Wed, Apr 19, 2023 at 7:37 PM Neal Cardwell <ncardwell=
>>>>>>> [e-mail]> wrote:
>>>>>>>
>>>>>>>> Working through examples for the "draft-ietf-tcpm-prr-rfc6937bis-03
>>>>>>>> and RecoverFS initialization" thread this evening, I ran into another
>>>>>>>> potential issue.
>>>>>>>>
>>>>>>>> The Linux TCP implementation of PRR explicitly/directly sets cwnd
>>>>>>>> to ssthresh at the end of fast recovery (in tcp_end_cwnd_reduction()). But
>>>>>>>> this behavior is not in the algorithm in the PRR RFC or draft, at least in
>>>>>>>> the figures in section 6, Algorithms. Maybe it is in the prose somewhere
>>>>>>>> and I missed it; but in that case I'd argue strongly to put this in the
>>>>>>>> figures in section 6, Algorithms.
>>>>>>>>
>>>>>>>> AFAICT in some cases this is strictly necessary to get cwnd to grow
>>>>>>>> to reach ssthresh. Without such a direct step, cwnd could end up far below
>>>>>>>> ssthresh at the end of recovery. Here's an example to illustrate:
>>>>>>>>
>>>>>>>> CC = CUBIC
>>>>>>>>
>>>>>>>> cwnd = 10
>>>>>>>>
>>>>>>>> The reordering degree was estimated to be large, so the connection
>>>>>>>> will wait for more than 3 packets to be SACKed before entering fast
>>>>>>>> recovery.
>>>>>>>>
>>>>>>>> --- Application writes 10*MSS.
>>>>>>>>
>>>>>>>> TCP sends packets P1 .. P10.
>>>>>>>> pipe = 10 packets in flight (P1 .. P10)
>>>>>>>>
>>>>>>>> --- P2..P9 SACKed  -> do nothing
>>>>>>>>
>>>>>>>> (Because the reordering degree was previously estimated to be
>>>>>>>> large.)
>>>>>>>>
>>>>>>>> --- P10 SACKed -> mark P1 as lost and enter fast recovery
>>>>>>>>
>>>>>>>> PRR:
>>>>>>>> ssthresh = CongCtrlAlg() = 7 packets // CUBIC
>>>>>>>> prr_delivered = 0
>>>>>>>> prr_out = 0
>>>>>>>> RecoverFS = snd.nxt - snd.una = 10 packets (P1..P10)
>>>>>>>>
>>>>>>>> DeliveredData = 1  (P10 was SACKed)
>>>>>>>>
>>>>>>>> prr_delivered += DeliveredData   ==> prr_delivered = 1
>>>>>>>>
>>>>>>>> pipe =  0  (all packets are SACKed or lost; P1 is lost, rest are
>>>>>>>> SACKed)
>>>>>>>>
>>>>>>>> safeACK = false (snd.una did not advance)
>>>>>>>>
>>>>>>>> if (pipe > ssthresh) => if (0 > 7) => false
>>>>>>>> else
>>>>>>>>   // PRR-CRB by default
>>>>>>>>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>>>>>>>>          = MAX(1 - 0, 1)
>>>>>>>>          = 1
>>>>>>>>
>>>>>>>>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>>>>>>>>          = MIN(7 - 0, 1)
>>>>>>>>          = 1
>>>>>>>>
>>>>>>>> cwnd = pipe + sndcnt
>>>>>>>>      = 0    + 1
>>>>>>>>      = 1
>>>>>>>>
>>>>>>>> retransmit P1
>>>>>>>>
>>>>>>>> prr_out += 1   ==> prr_out = 1
>>>>>>>>
>>>>>>>> --- P1 retransmit plugs hole; receive cumulative ACK for P1..P10
>>>>>>>>
>>>>>>>> DeliveredData = 1  (P1 was newly ACKed)
>>>>>>>>
>>>>>>>> prr_delivered += DeliveredData   ==> prr_delivered = 2
>>>>>>>>
>>>>>>>> pipe =  0  (all packets are cumuatively ACKed)
>>>>>>>>
>>>>>>>> safeACK = (snd.una advances and no further loss indicated)
>>>>>>>> safeACK = true
>>>>>>>>
>>>>>>>> if (pipe > ssthresh) => if (0 > 7) => false
>>>>>>>> else
>>>>>>>>   // PRR-CRB by default
>>>>>>>>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>>>>>>>>          = MAX(2 - 1, 1)
>>>>>>>>          = 1
>>>>>>>>   if (safeACK) => true
>>>>>>>>     // PRR-SSRB when recovery is in good progress
>>>>>>>>     sndcnt += 1   ==> sndcnt = 2
>>>>>>>>
>>>>>>>>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>>>>>>>>          = MIN(7 - 0, 2)
>>>>>>>>          = 2
>>>>>>>>
>>>>>>>> cwnd = pipe + sndcnt
>>>>>>>>      = 0    + 2
>>>>>>>>      = 2
>>>>>>>>
>>>>>>>> So we exit fast recovery with cwnd=2 even though ssthresh is 7.
>>>>>>>>
>>>>>>>> As noted above, the Linux TCP implementation does not suffer this
>>>>>>>> problem because it explicitly/directly sets cwnd to ssthresh at the end of
>>>>>>>> fast recovery.
>>>>>>>>
>>>>>>>> I would recommend including this cwnd=ssthresh step at the end of
>>>>>>>> recovery in the draft, to ensure that cwnd reaches ssthresh at the end of
>>>>>>>> fast recovery, even in cases like this where there will be insufficient
>>>>>>>> delivered data in fast recovery to allow pipe to incrementally grow to
>>>>>>>> reach ssthresh using PRR-SSRB.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> neal
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> tcpm mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/tcpm
>>>>>>>> <https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> tcpm mailing list
>>>>>>> [e-mail]
>>>>>>>
>>>>>>> https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw
>>>>>>>
>>>>>>>
>>>>>>> ------
>>>>>>> Randall Stewart
>>>>>>> [e-mail]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>