Re: [tcpm] draft-ietf-tcpm-prr-rfc6937bis-03: set cwnd to ssthresh exiting fast recovery?

[Randall Stewart <[e-mail]>]

Ahh I see.. In FreeBSD we are actually setting ssthresh at the entry to recovery like RFC6675, so when we
exit recovery we set cwnd to ssthresh. So we end up getting 1/2 Cwnd at the entry to recovery not the exit :)

And actually in one of our stacks we are using the <flight size at send> to cut in 1/2 which is even more conservative :)


R

> On May 2, 2023, at 2:31 PM, Neal Cardwell <[e-mail]> wrote:
> 
> Hi Yoshi,
> 
> You are right that because PRR always sets cwnd to ssthresh at the end of recovery, there will be some cases where with PRR cwnd jumps up drastically at the end of the recovery.
> 
> However, AFAIK cwnd jumping up drastically, per se, is not a problem. Big bursts of packets going into the network is a problem. And given the dynamics of the alternative loss recovery algorithms (RFC6675 and PRR), both can allow bursts of packets; just in different circumstances:
> 
> (1) RFC6675: Because RFC6675 sets cwnd once at the start of fast recovery, using (4.2) from RFC6675:
> 
> ssthresh = cwnd = (FlightSize / 2)
> 
> ...that means RFC6675 allows big bursts at the moment any loss is detected: any time L packets are lost, the sender can burst L more packets.
> 
> (2) PRR: PRR is specifically designed to avoid big bursts in response to packet losses; no matter the structure or timing of the losses, PRR only allows a big burst at the end of Fast Recovery after all holes have been plugged, and the algorithm sets cwnd to ssthresh.
> 
> So in your example ("For example, many packets were lost before entering recovery"), AFAICT RFC6675 can allow a big burst at the beginning of recovery, when the lost packets are detected. AFAICT in this case PRR can allow a burst of packets at the end of recovery when it sets cwnd to ssthresh, but at least at this point the bottleneck queue has potentially drained somewhat.
> 
> Please let me know if that analysis misses something important. :-)
> 
> Thanks!
> neal
> 
> 
> On Mon, May 1, 2023 at 5:22 PM Yoshifumi Nishida <[e-mail] <mailto:[e-mail]>> wrote:
> Hi Randall,
> 
> I might miss something, but here's what I've thought..
> If we lost many packets in a RTT such as the Figure 5 in the 6937bis draft, I think the window growth during the recovery period will be bound by PRR-CRB or PRR-SSRB.
> Hence, I think the cwnd at the end of recovery can be smaller than we expect as shown in figure 5. 
> --
> Yoshi
> 
> On Mon, May 1, 2023 at 4:17 AM Randall Stewart <[e-mail] <mailto:[e-mail]>> wrote:
> Hi Neal and Yoshi:
> 
> Neal: So the FreeBSD implementation in rack, like linux, does the same exact thing set cwnd to ssthresh at
> exit from recovery. 
> 
> Yoshi: I don’t see how this would cause cwnd to be larger, since at the entry to recovery you set ssthresh = cwnd *  Beta. But
>           maybe I am missing something, can you give an example like Neal did below?
> 
> 
> Thanks
> 
> R
> 
>> On May 1, 2023, at 5:32 AM, Yoshifumi Nishida <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>> Hi Neal,
>> 
>> If we always set cwnd to ssthresh at the end of recovery, I am guessing there will be some cases where cwnd jumps up drastically at the end of the recovery. For example, many packets were lost before entering recovery.  Or, am I missing something?
>> --
>> Yoshi
>> 
>> On Wed, Apr 19, 2023 at 7:37 PM Neal Cardwell <ncardwell=[e-mail] <mailto:[e-mail]>> wrote:
>> Working through examples for the "draft-ietf-tcpm-prr-rfc6937bis-03 and RecoverFS initialization" thread this evening, I ran into another potential issue.
>> 
>> The Linux TCP implementation of PRR explicitly/directly sets cwnd to ssthresh at the end of fast recovery (in tcp_end_cwnd_reduction()). But this behavior is not in the algorithm in the PRR RFC or draft, at least in the figures in section 6, Algorithms. Maybe it is in the prose somewhere and I missed it; but in that case I'd argue strongly to put this in the figures in section 6, Algorithms.
>> 
>> AFAICT in some cases this is strictly necessary to get cwnd to grow to reach ssthresh. Without such a direct step, cwnd could end up far below ssthresh at the end of recovery. Here's an example to illustrate:
>> 
>> CC = CUBIC
>> 
>> cwnd = 10
>> 
>> The reordering degree was estimated to be large, so the connection will wait for more than 3 packets to be SACKed before entering fast recovery.
>> 
>> --- Application writes 10*MSS.
>> 
>> TCP sends packets P1 .. P10.
>> pipe = 10 packets in flight (P1 .. P10)
>> 
>> --- P2..P9 SACKed  -> do nothing
>> 
>> (Because the reordering degree was previously estimated to be large.)
>> 
>> --- P10 SACKed -> mark P1 as lost and enter fast recovery
>> 
>> PRR:
>> ssthresh = CongCtrlAlg() = 7 packets // CUBIC
>> prr_delivered = 0
>> prr_out = 0
>> RecoverFS = snd.nxt - snd.una = 10 packets (P1..P10)
>> 
>> DeliveredData = 1  (P10 was SACKed)
>> 
>> prr_delivered += DeliveredData   ==> prr_delivered = 1
>> 
>> pipe =  0  (all packets are SACKed or lost; P1 is lost, rest are SACKed)
>> 
>> safeACK = false (snd.una did not advance)
>> 
>> if (pipe > ssthresh) => if (0 > 7) => false
>> else
>>   // PRR-CRB by default
>>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>>          = MAX(1 - 0, 1)
>>          = 1
>> 
>>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>>          = MIN(7 - 0, 1)
>>          = 1
>> 
>> cwnd = pipe + sndcnt
>>      = 0    + 1
>>      = 1
>> 
>> retransmit P1
>> 
>> prr_out += 1   ==> prr_out = 1
>> 
>> --- P1 retransmit plugs hole; receive cumulative ACK for P1..P10
>> 
>> DeliveredData = 1  (P1 was newly ACKed)
>> 
>> prr_delivered += DeliveredData   ==> prr_delivered = 2
>> 
>> pipe =  0  (all packets are cumuatively ACKed)
>> 
>> safeACK = (snd.una advances and no further loss indicated)
>> safeACK = true
>> 
>> if (pipe > ssthresh) => if (0 > 7) => false
>> else
>>   // PRR-CRB by default
>>   sndcnt = MAX(prr_delivered - prr_out, DeliveredData)
>>          = MAX(2 - 1, 1)
>>          = 1
>>   if (safeACK) => true
>>     // PRR-SSRB when recovery is in good progress
>>     sndcnt += 1   ==> sndcnt = 2
>> 
>>   sndcnt = MIN(ssthresh - pipe, sndcnt)
>>          = MIN(7 - 0, 2)
>>          = 2
>> 
>> cwnd = pipe + sndcnt
>>      = 0    + 2
>>      = 2
>> 
>> So we exit fast recovery with cwnd=2 even though ssthresh is 7.
>> 
>> As noted above, the Linux TCP implementation does not suffer this problem because it explicitly/directly sets cwnd to ssthresh at the end of fast recovery.
>> 
>> I would recommend including this cwnd=ssthresh step at the end of recovery in the draft, to ensure that cwnd reaches ssthresh at the end of fast recovery, even in cases like this where there will be insufficient delivered data in fast recovery to allow pipe to incrementally grow to reach ssthresh using PRR-SSRB.
>> 
>> Best regards,
>> neal
>> 
>> _______________________________________________
>> tcpm mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/tcpm <https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://www.ietf.org/mailman/listinfo/tcpm%26source%3Dgmail-imap%26ust%3D1683538345000000%26usg%3DAOvVaw2cOITQpYcuP_M95396rEmw&source=gmail-imap&ust=1683657088000000&usg=AOvVaw3x8_wWDqG-26ZiCQcMxCrV>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.google.com/url?q=https://www.ietf.org/mailman/listinfo/tcpm&source=gmail-imap&ust=1683538345000000&usg=AOvVaw2cOITQpYcuP_M95396rEmw <https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://www.ietf.org/mailman/listinfo/tcpm%26source%3Dgmail-imap%26ust%3D1683538345000000%26usg%3DAOvVaw2cOITQpYcuP_M95396rEmw&source=gmail-imap&ust=1683657088000000&usg=AOvVaw3x8_wWDqG-26ZiCQcMxCrV>
> 
> ------
> Randall Stewart
> [e-mail] <mailto:[e-mail]>
> 
> 
> 

------
Randall Stewart
[e-mail]