Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set cwnd to ssthresh exiting fast recovery?

[Yuchung Cheng <[e-mail]>]

Hi Yoshifumi,

That part is how the "RecoverFS" state variable is calculated in the draft.
See the diff of 03/04 on Section 5 and 6 regarding "RecoverFS" state
variable definition and computation.
https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-prr-rfc6937bis-04

Does that make sense?

On Tue, Aug 8, 2023 at 12:01 AM Yoshifumi Nishida <[e-mail]>
wrote:

> Hi Yuchung,
>
> I think you have already updated the draft on the following point from the
> discussions in the last WG meeting.
> Could you point out which part has been updated? I'm just checking..
> Thanks,
> --
> Yoshi
>
> On Fri, May 5, 2023 at 11:51 AM Yoshifumi Nishida <[e-mail]>
> wrote:
>
>> Hi Neal,
>>
>> Yes, I think I understand your point.
>> I prefer the current logic in some ways as it's more conservative as I
>> think we cannot always presume that queue has been drained at the end of
>> recovery.
>> But, I also think it may look too conservative.
>> I am expecting that the authors provide some insights on this point.
>> --
>> Yoshi
>>
>>
>> On Tue, May 2, 2023 at 11:31 AM Neal Cardwell <[e-mail]>
>> wrote:
>>
>>> Hi Yoshi,
>>>
>>> You are right that because PRR always sets cwnd to ssthresh at the end
>>> of recovery, there will be some cases where with PRR cwnd jumps up
>>> drastically at the end of the recovery.
>>>
>>> However, AFAIK cwnd jumping up drastically, per se, is not a problem.
>>> Big bursts of packets going into the network is a problem. And given the
>>> dynamics of the alternative loss recovery algorithms (RFC6675 and PRR),
>>> both can allow bursts of packets; just in different circumstances:
>>>
>>> (1) RFC6675: Because RFC6675 sets cwnd once at the start of fast
>>> recovery, using (4.2) from RFC6675:
>>>
>>> ssthresh = cwnd = (FlightSize / 2)
>>>
>>> ...that means RFC6675 allows big bursts at the moment any loss is
>>> detected: any time L packets are lost, the sender can burst L more packets.
>>>
>>> (2) PRR: PRR is specifically designed to avoid big bursts in response to
>>> packet losses; no matter the structure or timing of the losses, PRR only
>>> allows a big burst at the end of Fast Recovery after all holes have been
>>> plugged, and the algorithm sets cwnd to ssthresh.
>>>
>>> So in your example ("For example, many packets were lost before entering
>>> recovery"), AFAICT RFC6675 can allow a big burst at the beginning of
>>> recovery, when the lost packets are detected. AFAICT in this case PRR can
>>> allow a burst of packets at the end of recovery when it sets cwnd to
>>> ssthresh, but at least at this point the bottleneck queue has potentially
>>> drained somewhat.
>>>
>>> Please let me know if that analysis misses something important. :-)
>>>
>>> Thanks!
>>> neal
>>>
>>>
>>> On Mon, May 1, 2023 at 5:22 PM Yoshifumi Nishida <[e-mail]>
>>> wrote:
>>>
>>>> Hi Randall,
>>>>
>>>> I might miss something, but here's what I've thought..
>>>> If we lost many packets in a RTT such as the Figure 5 in the 6937bis
>>>> draft, I think the window growth during the recovery period will be bound
>>>> by PRR-CRB or PRR-SSRB.
>>>> Hence, I think the cwnd at the end of recovery can be smaller than we
>>>> expect as shown in figure 5.
>>>> --
>>>> Yoshi
>>>>
>>>> On Mon, May 1, 2023 at 4:17 AM Randall Stewart <[e-mail]> wrote:
>>>>
>>>>> Hi Neal and Yoshi:
>>>>>
>>>>> Neal: So the FreeBSD implementation in rack, like linux, does the same
>>>>> exact thing set cwnd to ssthresh at
>>>>> exit from recovery.
>>>>>
>>>>> Yoshi: I don’t see how this would cause cwnd to be larger, since at
>>>>> the entry to recovery you set ssthresh = cwnd *  Beta. But
>>>>>           maybe I am missing something, can you give an example like
>>>>> Neal did below?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> R
>>>>>
>>>>> On May 1, 2023, at 5:32 AM, Yoshifumi Nishida <[e-mail]>
>>>>> wrote:
>>>>>
>>>>> Hi Neal,
>>>>>
>>>>> If we always set cwnd to ssthresh at the end of recovery, I am
>>>>> guessing there will be some cases where cwnd jumps up drastically at the
>>>>> end of the recovery. For example, many packets were lost before entering
>>>>> recovery.  Or, am I missing something?
>>>>> --
>>>>> Yoshi
>>>>>
>>>>> On Wed, Apr 19, 2023 at 7:37 PM Neal Cardwell <ncardwell=
>>>>> [e-mail]> wrote:
>>>>>
>>>>>> Working through examples for the "draft-ietf-tcpm-prr-rfc6937bis-03
>>>>>> and RecoverFS initialization" thread this evening, I ran into another
>>>>>> potential issue.
>>>>>>
>>>>>> The Linux TCP implementation of PRR explicitly/directly sets cwnd to
>>>>>> ssthresh at the end of fast recovery (in tcp_end_cwnd_reduction()). But
>>>>>> this behavior is not in the algorithm in the PRR RFC or draft, at least in
>>>>>> the figures in section 6, Algorithms. Maybe it is in the prose somewhere
>>>>>> and I missed it; but in that case I'd argue strongly to put this in the
>>>>>> figures in section 6, Algorithms.
>>>>>>
>>>>>> AFAICT in some cases this is strictly necessary to get cwnd to grow
>>>>>> to reach ssthresh. Without such a direct step, cwnd could end up far below
>>>>>> ssthresh at the end of recovery. Here's an example to illustrate:
>>>>>>
>>>>>> CC = CUBIC
>>>>>>
>>>>>> cwnd = 10
>>>>>>
>>>>>> The reordering degree was estimated to be large, so the connection
>>>>>> will wait for more than 3 packets to be SACKed before entering fast
>>>>>> recovery.
>>>>>>
>>>>>> --- Application writes 10*MSS.
>>>>>>
>>>>>> TCP sends packets P1 .. P10.
>>>>>> pipe = 10 packets in flight (P1 .. P10)
>>>>>>
>>>>>> --- P2..P9 SACKed  -> do nothing
>>>>>>
>>>>>> (Because the reordering degree was previously estimated to be large.)
>>>>>>
>>>>>> --- P10 SACKed -> mark P1 as lost and enter fast recovery
>>>>>>
>>>>>> PRR:
>>>>>> ssthresh = CongCtrlAlg() = 7 packets // CUBIC
>>>>>> prr_delivered = 0
>>>>>> prr_out = 0
>>>>>> RecoverFS = snd.nxt - snd.una = 10 packets (P1..P10)
>>>>>>
>>>>>> DeliveredData = 1  (P10 was SACKed)
>>>>>>
>>>>>> prr_delivered += DeliveredData   ==> prr_delivered = 1
>>>>>>
>>>>>> pipe =  0  (all packets are SACKed or lost; P1 is lost, rest are
>>>>>> SACKed)
>>>>>>
>>>>>> safeACK = false (snd.una did not advance)
>>>>>>
>>>>>> if (pipe > ssthresh) => if (0 > 7) => false
>>>>>> else
>>>>>>   // PRR-CRB by default
>>>>>>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>>>>>>          = MAX(1 - 0, 1)
>>>>>>          = 1
>>>>>>
>>>>>>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>>>>>>          = MIN(7 - 0, 1)
>>>>>>          = 1
>>>>>>
>>>>>> cwnd = pipe + sndcnt
>>>>>>      = 0    + 1
>>>>>>      = 1
>>>>>>
>>>>>> retransmit P1
>>>>>>
>>>>>> prr_out += 1   ==> prr_out = 1
>>>>>>
>>>>>> --- P1 retransmit plugs hole; receive cumulative ACK for P1..P10
>>>>>>
>>>>>> DeliveredData = 1  (P1 was newly ACKed)
>>>>>>
>>>>>> prr_delivered += DeliveredData   ==> prr_delivered = 2
>>>>>>
>>>>>> pipe =  0  (all packets are cumuatively ACKed)
>>>>>>
>>>>>> safeACK = (snd.una advances and no further loss indicated)
>>>>>> safeACK = true
>>>>>>
>>>>>> if (pipe > ssthresh) => if (0 > 7) => false
>>>>>> else
>>>>>>   // PRR-CRB by default
>>>>>>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>>>>>>          = MAX(2 - 1, 1)
>>>>>>          = 1
>>>>>>   if (safeACK) => true
>>>>>>     // PRR-SSRB when recovery is in good progress
>>>>>>     sndcnt += 1   ==> sndcnt = 2
>>>>>>
>>>>>>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>>>>>>          = MIN(7 - 0, 2)
>>>>>>          = 2
>>>>>>
>>>>>> cwnd = pipe + sndcnt
>>>>>>      = 0    + 2
>>>>>>      = 2
>>>>>>
>>>>>> So we exit fast recovery with cwnd=2 even though ssthresh is 7.
>>>>>>
>>>>>> As noted above, the Linux TCP implementation does not suffer this
>>>>>> problem because it explicitly/directly sets cwnd to ssthresh at the end of
>>>>>> fast recovery.
>>>>>>
>>>>>> I would recommend including this cwnd=ssthresh step at the end of
>>>>>> recovery in the draft, to ensure that cwnd reaches ssthresh at the end of
>>>>>> fast recovery, even in cases like this where there will be insufficient
>>>>>> delivered data in fast recovery to allow pipe to incrementally grow to
>>>>>> reach ssthresh using PRR-SSRB.
>>>>>>
>>>>>> Best regards,
>>>>>> neal
>>>>>>
>>>>>> _______________________________________________
>>>>>> tcpm mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/tcpm
>>>>>> <https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw>
>>>>>>
>>>>> _______________________________________________
>>>>> tcpm mailing list
>>>>> [e-mail]
>>>>>
>>>>> https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw
>>>>>
>>>>>
>>>>> ------
>>>>> Randall Stewart
>>>>> [e-mail]
>>>>>
>>>>>
>>>>>
>>>>>