Re: [tcpm] Hystart survey of large server operators

[Yuchung Cheng <[e-mail]>]

On Thu, Jul 29, 2021 at 9:28 AM Bob Briscoe <[e-mail]> wrote:

> Yuchung,
>
> On 29/07/2021 00:35, Yuchung Cheng wrote:
>
> same experience with Chrisitian
>
> For more than half a decade Google traffic used Linux CUBIC Hystart. This
> includes google.com and YouTube public Internet traffic, as well as
> internal Google TCP traffic. When we deployed pacing around 2013 we
> disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as
> it causes false SS exit. But we continued to use the stock Hystart Delay
> mechanism. ~2014-15 we switched to BBR which uses a different startup more
> robust to delay jitters.
>
>
> [BB] You or Neal (sry can't remember which) were one of those I surveyed
> who said they had disabled Hystart. So perhaps this was a misunderstanding
> over the ACK-train part.
>
Yes this must be mis-communication. Since that was in 2018, it's either "we
don't use hystart b/c BBR doesn't use hystart" or "we don't use hystart
ack-train when hystart was used in Cubic before BBR".



> But IMO the simpler hystart mitigating SS over-shoot is still highly
> valuable for Cubic and other congestion controls, as Internet bandwidth
> continues to hike.
>
> Did those providers disable hystart b/c of its poor interaction between
> the ack-train mode and pacing?
>
>
> [BB] I didn't ask them why.
>
>
>
> Bob
>
> Maybe it's time to disable the ack-train approach in the upstream
> Linux when hystart++ is standardized.
>
> On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <[e-mail]>
> wrote:
>
>> I certainly do not qualify as a "large operator", but when implementing
>> Cubic for QUIC I found that Hystart was critical for performance.
>> Specifically, the classical slow-start is very prone to overshooting the
>> capacity of the path, which causes large batches of errors. With web-like
>> traffic, these batches of errors cause increase latency for the first
>> transactions in a session, and thus a drop in perceived quality. Hystart
>> solves that. With Hystart, I observed that a large fraction of sessions do
>> not experience any packet loss at all.
>>
>> As an aside, we should strive for this suppression of packet losses.
>> That's actually a big reason for moving from Cubic to BBR. Hystart
>> suppresses the losses during the initial phase of the connection, but Cubic
>> still relies on periodically testing the limit of path capacity and causing
>> losses during the subsequent phase. BBR's probing for bottleneck bandwidth
>> is much more conservative, does not cause such losses. It might be possible
>> to adapt Cubic to also not cause losses, for example by ending an epoch
>> early if too many CE marks are received or if the RTT increases. That would
>> be worth trying.
>>
>> -- Christian Huitema
>> On 7/28/2021 3:19 AM, Bob Briscoe wrote:
>>
>> Yuchung,
>>
>> It was during a Mar 2018 ad hoc workshop Jana had organized at the London
>> IETF entitled 'BBR and the intersection with other work". I can't remember
>> why I needed to know at the time, but during the break I approached the
>> major server operators individually, established whether they used Cubic,
>> and if so asked whether they used Hystart or disabled it. It's hard to
>> anonymize the results, because IMMSMC all those that used Cubic said they
>> disabled Hystart.
>>
>> If this isn't correct, then maybe people who replied at the time thought
>> they were being asked something else. Or maybe it was correct then but
>> isn't now.
>>
>>
>> Bob
>>
>>
>> On 28/07/2021 00:46, Yuchung Cheng wrote:
>>
>> Wait -- how is hystart "invariably disabled" in Linux (cubic)!?
>>
>> What data indicates that
>>
>> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <[e-mail]
>> <mailto:[e-mail]> <[e-mail]>> wrote:
>>
>>     Any large server operator out there who are using Cubic,
>>     If you're willing to state whether or not your operations disable
>>     Hystart, pls do so in reply.
>>     Then Praveen can cite this mailing list thread in the Hystart++
>>     draft, as requested below.
>>
>>     If you are willing to reply privately, I would be willing to keep
>>     your confidences, and provide an anonymized result to the list.
>>
>>     Cheers
>>
>>
>>
>>     Bob
>>
>>     On 27/07/2021 01:50, Praveen Balasubramanian wrote:
>>
>>
>>     Although Hystart is default enabled in Linux, it is invariably
>>     disabled. So, it's misleading to just say Hystart is default
>>     enabled, which implies it's widely used, when people clearly find
>>     it has problems (which motivates Hystart++). I found this out
>>     through an informal survey I did at the Mar'18 IETF in London by
>>     asking round the implementers of the most prevalent stacks (I
>>     would name names if I could find the note I later sent to someone
>>     or to some list, but I can't find it - sry).
>>
>>
>>     I'd like some citations on this versus anecdata if possible
>>     before I add that caveat to the text. Do large deployments
>>     disable this? I haven't come across this suggestion in any
>>     Linux tuning guides to date.
>>
>>
>>     --
>> ________________________________________________________________
>>     Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>
>> <http://bobbriscoe.net/>
>>
>>     _______________________________________________
>>     tcpm mailing list
>>     [e-mail] <mailto:[e-mail]> <[e-mail]>
>>     https://www.ietf.org/mailman/listinfo/tcpm
>>     <https://www.ietf.org/mailman/listinfo/tcpm>
>> <https://www.ietf.org/mailman/listinfo/tcpm>
>>
>>
>>
>> _______________________________________________
>> tcpm mailing [e-mail]://www.ietf.org/mailman/listinfo/tcpm
>>
>>
> --
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>
>