Re: [tcpm] [EXTERNAL] Re: hystart++

[Yi Huang <[e-mail]>]

Hi Neal and Randall,



We implemented the suggested "set cssBaseLineMinRTT to (lastRoundMinRtt + RttThresh)" and we did an A/B test over our own emulated WAN test cases. We noticed that there's a regression in scenarios with latency jitters as you can see in the below graph. Mitigating spurious Hystart exit was why we invented CSS. Based on these results we concluded that the current draft algorithm is the better choice.



This is how we define test cases:

100ms,100Mbps,1250pkt,64000B,2000,50jitms,5jitdiv

=>

100ms RTT, 100Mbps bottleneck bandwidth, bottleneck queue length 1250 packets, IO size 64KB, IO count 2000, 50ms latency jitter, 1/5 jitter probability per packet





[cid:a108063c-37fa-402f-b920-c116fab108e9]



>> One option would be to track the maximum number of packets delivered in a round of slow-start (slowStartMaxDeliveredPackets), and only check for spurious Hystart++ exits if the most recent round of >> CSS has at least slowStartMaxDeliveredPackets delivered. Something like:

>>- if (currentRoundDeliveredPackets >= slowStartMaxDeliveredPackets &&

>>      currentRoundMinRTT < cssBaselineMinRtt)

>>

>>         o  cssBaselineMinRtt = infinity

>>

>>         o  resume slow start including HyStart++



We have not tried this but it requires rounds to be well aligned with RTTs. E.g. if one round takes about 2 RTTs and the other round takes 0.5 RTT, it won't consistently work.



Thanks,



Yi

________________________________
From: tcpm <[e-mail]> on behalf of Randall Stewart <rrs=[e-mail]>
Sent: Thursday, December 2, 2021 12:08 PM
To: Neal Cardwell <ncardwell=[e-mail]>
Cc: [e-mail] <[e-mail]>; Randall Stewart <rrs=[e-mail]>
Subject: [EXTERNAL] Re: [tcpm] hystart++

[Some people who received this message don't often get email from rrs=[e-mail]. Learn why this is important at http://aka.ms/LearnAboutSenderIdentification.]

Neal:

Great observation!!

> On Dec 2, 2021, at 2:43 PM, Neal Cardwell <ncardwell=[e-mail]> wrote:
>
> Thanks, Randall. That's a nice analysis, and I agree that your proposal sounds like a nice improvement.
>
> This nice analysis brings to mind another potential (related) issue that might cause spurious oscillations: after Hystart++ exits slow-start, if there are application-limited periods where the amount of data in flight falls below the BDP (e.g. flow goes idle and then restarts), then we would expect the currentRoundMinRTT to be very low, triggering a resumption of slow-start in the current logic, even if the Hystart++ exit of slow-start was correct.
>

I have actually seen this occur.. in fact I was wondering how I was going to test
dropping back out in my lab (was thinking packet drill) and then my logging noted
that it had happened.. and it was because nginx had taken a 25ms break in applying
data .. RTT was 20ms so the whole network went idle by the time the buffer was refilled and
thus a bounce out and back in later :)

> It seems that the connection should not use a data ACK's RTT sample to estimate whether Hystart++ exit was spurious unless the amount of data in flight at the time that data was sent was "sufficiently large". How to define "sufficiently large" may be a little tricky (the cwnd is probably not a good gauge for "sufficiently large" since it is probably somewhere out beyond twice the BDP in CSS).
>

Yeah I was wondering if we might not want to adjust exit based on some other
metric in app limited cases...

> One option would be to track the maximum number of packets delivered in a round of slow-start (slowStartMaxDeliveredPackets), and only check for spurious Hystart++ exits if the most recent round of CSS has at least slowStartMaxDeliveredPackets delivered. Something like:
>
> - if (currentRoundDeliveredPackets >= slowStartMaxDeliveredPackets &&
>       currentRoundMinRTT < cssBaselineMinRtt)
>
>          o  cssBaselineMinRtt = infinity
>
>          o  resume slow start including HyStart++


That sounds like a very good adjustment.. I will maybe prototype it in my
lab and see if I can recreate the bounce I saw and see how well it works ;)

Best

R

>
> best regards,
> neal
>
>
>
>
>
>
> On Thu, Dec 2, 2021 at 1:46 PM Randall Stewart <rrs=[e-mail]> wrote:
> Greetings all:
>
> I have implemented hystart++ off of the draft-03 for FreeBSD
> (both newreno and cubic) and I have noted one thing that I think
> is a bug in the spec:
>
> Draft-03 currently says:
>
>
>
>         o  if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh))
>
>            +  cssBaselineMinRtt = currentRoundMinRTT
>
>            +  exit slow start and enter CSS
>
>
> <And>
>
>
>
>      -  if (currentRoundMinRTT < cssBaselineMinRtt)
>
>         o  cssBaselineMinRtt = infinity
>
>         o  resume slow start including HyStart++
>
>
>
> But notice that the threshold for entering CSS is (lastRoundMinRtt + RttThresh) and
> the exiting of CSS back to Slow Start is (currentRoundMinRTT < cssBaseLineMinRTT), which can
> under the right circumstances set up an oscillation. In the case I observed I saw lastRoundMinRtt set
> to like 63ms and currentRoundMinRtt set at 115ms. And the next RTT in was 114ms.. so out of
> CSS into Slow Start, but the next measurement (114ms again) pushes you back into CSS since
> the lastRoundMinRTT + RttThresh is around 63ms or so.
>
> I would think what you really want here is to set cssBaseLineMinRTT to (lastRoundMinRtt + RttThresh).
>
> I.e. on entry:
>
>
>         o  if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh))
>
>            +  cssBaselineMinRtt = (lastRoundMinRtt + RttThresh)
>
>            +  exit slow start and enter CSS
>
> R
>
> ------
> Randall Stewart
> [e-mail]
>
>
>
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=04%7C01%7Chuanyi%40microsoft.com%7Cee82df6dc2ae4362922d08d9b5cf9fae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637740726320069237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=1oNiudKBKYJCz50wNxM30alScXlGJDZ2QsSSH1YBqgs%3D&amp;reserved=0
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Furl%3Fq%3Dhttps%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm%26source%3Dgmail-imap%26ust%3D1639079070000000%26usg%3DAOvVaw1S8xcmYu7Uv0BTPfUSlc5Z&amp;data=04%7C01%7Chuanyi%40microsoft.com%7Cee82df6dc2ae4362922d08d9b5cf9fae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637740726320069237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hMIondYgA%2B6EMrvwjS6RzmnLy%2B8OKFN6WftwPmoq8tk%3D&amp;reserved=0

------
Randall Stewart
[e-mail]