Re: [tcpm] Hystart survey of large server operators

Christian Huitema <huitema@huitema.net> Wed, 28 July 2021 16:47 UTC

Return-Path: <huitema@huitema.net>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 466AE3A1890 for <tcpm@ietfa.amsl.com>; Wed, 28 Jul 2021 09:47:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.888
X-Spam-Level:
X-Spam-Status: No, score=-1.888 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xfO5eC7_bUcM for <tcpm@ietfa.amsl.com>; Wed, 28 Jul 2021 09:47:21 -0700 (PDT)
Received: from mx36-out21.antispamcloud.com (mx36-out21.antispamcloud.com [209.126.121.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C83013A17FB for <tcpm@ietf.org>; Wed, 28 Jul 2021 09:47:04 -0700 (PDT)
Received: from xse109.mail2web.com ([66.113.196.109] helo=xse.mail2web.com) by mx133.antispamcloud.com with esmtp (Exim 4.92) (envelope-from <huitema@huitema.net>) id 1m8mhu-000ZSO-5i for tcpm@ietf.org; Wed, 28 Jul 2021 18:47:01 +0200
Received: from xsmtp22.mail2web.com (unknown [10.100.68.61]) by xse.mail2web.com (Postfix) with ESMTPS id 4GZfhF52SXzLDp for <tcpm@ietf.org>; Wed, 28 Jul 2021 09:46:53 -0700 (PDT)
Received: from [10.5.2.14] (helo=xmail04.myhosting.com) by xsmtp22.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from <huitema@huitema.net>) id 1m8mhp-0007Ha-In for tcpm@ietf.org; Wed, 28 Jul 2021 09:46:53 -0700
Received: (qmail 30611 invoked from network); 28 Jul 2021 16:46:52 -0000
Received: from unknown (HELO [192.168.1.104]) (Authenticated-user:_huitema@huitema.net@[172.58.43.253]) (envelope-sender <huitema@huitema.net>) by xmail04.myhosting.com (qmail-ldap-1.03) with ESMTPA for <draft-ietf-tcpm-hystartplusplus@ietf.org>; 28 Jul 2021 16:46:52 -0000
To: Bob Briscoe <ietf@bobbriscoe.net>, Yuchung Cheng <ycheng@google.com>
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, "draft-ietf-tcpm-hystartplusplus@ietf.org" <draft-ietf-tcpm-hystartplusplus@ietf.org>
References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <PH0PR00MB1030126A3220BC056A406490B6E99@PH0PR00MB1030.namprd00.prod.outlook.com> <PH0PR00MB1030E0697BE8E93074B901D2B6E99@PH0PR00MB1030.namprd00.prod.outlook.com> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <CAK6E8=f7qKDs-MFr4G6bpz82Swn3iCJEkWLL5yr+vV8z9zMb=Q@mail.gmail.com> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net>
From: Christian Huitema <huitema@huitema.net>
Message-ID: <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net>
Date: Wed, 28 Jul 2021 09:46:52 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0
MIME-Version: 1.0
In-Reply-To: <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net>
Content-Type: multipart/alternative; boundary="------------1455D1D7C3FC1353F392291E"
Content-Language: en-US
X-Originating-IP: 66.113.196.109
X-Spampanel-Domain: xsmtpout.mail2web.com
X-Spampanel-Username: 66.113.196.0/24
Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.196.0/24@xsmtpout.mail2web.com
X-Spampanel-Outgoing-Class: unsure
X-Spampanel-Outgoing-Evidence: Combined (0.15)
X-Recommended-Action: accept
X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT9WLQux0N3HQm8ltz8rnu+BPUtbdvnXkggZ 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5yjTC8Fov82/EJuxz8FihBPKj/EwzSHE5FGYwwjsNRPCNsk 4s3QUE9Ikbu0D7c8RPbmD6wdmZPcItWbGe10hXJtXL4FsauCVkDjmcYJdU3yWp7KuHNaaKdg7iBE ZefdsNUFWKwa/wzJUjmazeC7Imcapebr0kNyYC289u5HlaNj1BQ6V51u76v35b1wNe/MvdKAGdwU TZKlze5ERymXAD3v2+J9PgaoF8SQHto3le4zsHTaeQtlKubP6iUTjj6yPARK6buALVaA782LKxg6 vRmng8N1aLhXqdc+jC1RcnVud53D5caUhbVtvqItBqoizkEt9O20UjkwI0v+LOlw05G4BS+iyyNq bT8dUMXMJ4tUCMj6G37ZfAMLceP5aNHPt26RBupu5v1nytoNnc138GfEJRQ2qC7jjynPIHPNqSn4 QTXUjLjYWQt1/5xnQymMoPsgr/U0flMcy2Vi/IcBgY4arPaiJ1W6hAyiRC61jekdwIcXNugoOEbH RyFULpSjm7jZ1h/HfDRQ5Ig8VhPsPE8NaP2gA77cO7WeI9Ftai6fusDXp7t2nLqCQxvxn0YNWXvz HU+t5Fa1CSbFsnUGqUzJDRojSVizNl0ce/s7u0P9b9Tml6eOMCV9kYYwkPx6ZsXvIUzTXkDAiiJi mGhLUFuS2lhaIetXfCg1JdAVrOwKfDfxVmiprXIZY6X8fH9rAIlUoFIvD3sIcP1fhJPM6B/87s2a Xm69McMIC42KTw7i32Lua3RJ0N4pwp1zcex/7jiJ+dym1L8cD17Js0v4cp1MaDI3VMeXrtWilILQ zgNtADcKVNeVJ9BXyu9+ceCqThTYg2px1fSoqxQCCHnLMo/m9VKh99btUAanjnMCAH2co+fBoeG+ Hs0afhsY/5zhNYWRVYKU9W9tbmVXJBqdHHDmZEKhyNAv1N35kYWaEdgLurFV5oTvAcwA4rM3FkfW 8/1kE/e7sUnsVpINvARNxpFO
X-Report-Abuse-To: spam@quarantine11.antispamcloud.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/8OGy4YTLjIN7bQdzwArVNr9ZxDQ>
Subject: Re: [tcpm] Hystart survey of large server operators
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 28 Jul 2021 16:47:33 -0000

I certainly do not qualify as a "large operator", but when implementing 
Cubic for QUIC I found that Hystart was critical for performance. 
Specifically, the classical slow-start is very prone to overshooting the 
capacity of the path, which causes large batches of errors. With 
web-like traffic, these batches of errors cause increase latency for the 
first transactions in a session, and thus a drop in perceived quality. 
Hystart solves that. With Hystart, I observed that a large fraction of 
sessions do not experience any packet loss at all.

As an aside, we should strive for this suppression of packet losses. 
That's actually a big reason for moving from Cubic to BBR. Hystart 
suppresses the losses during the initial phase of the connection, but 
Cubic still relies on periodically testing the limit of path capacity 
and causing losses during the subsequent phase. BBR's probing for 
bottleneck bandwidth is much more conservative, does not cause such 
losses. It might be possible to adapt Cubic to also not cause losses, 
for example by ending an epoch early if too many CE marks are received 
or if the RTT increases. That would be worth trying.

-- Christian Huitema

On 7/28/2021 3:19 AM, Bob Briscoe wrote:
> Yuchung,
>
> It was during a Mar 2018 ad hoc workshop Jana had organized at the 
> London IETF entitled 'BBR and the intersection with other work". I 
> can't remember why I needed to know at the time, but during the break 
> I approached the major server operators individually, established 
> whether they used Cubic, and if so asked whether they used Hystart or 
> disabled it. It's hard to anonymize the results, because IMMSMC all 
> those that used Cubic said they disabled Hystart.
>
> If this isn't correct, then maybe people who replied at the time 
> thought they were being asked something else. Or maybe it was correct 
> then but isn't now.
>
>
> Bob
>
>
> On 28/07/2021 00:46, Yuchung Cheng wrote:
>> Wait -- how is hystart "invariably disabled" in Linux (cubic)!?
>>
>> What data indicates that
>>
>> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net 
>> <mailto:ietf@bobbriscoe.net>> wrote:
>>
>>     Any large server operator out there who are using Cubic,
>>     If you're willing to state whether or not your operations disable
>>     Hystart, pls do so in reply.
>>     Then Praveen can cite this mailing list thread in the Hystart++
>>     draft, as requested below.
>>
>>     If you are willing to reply privately, I would be willing to keep
>>     your confidences, and provide an anonymized result to the list.
>>
>>     Cheers
>>
>>
>>
>>     Bob
>>
>>     On 27/07/2021 01:50, Praveen Balasubramanian wrote:
>>>
>>>     Although Hystart is default enabled in Linux, it is invariably
>>>     disabled. So, it's misleading to just say Hystart is default
>>>     enabled, which implies it's widely used, when people clearly find
>>>     it has problems (which motivates Hystart++). I found this out
>>>     through an informal survey I did at the Mar'18 IETF in London by
>>>     asking round the implementers of the most prevalent stacks (I
>>>     would name names if I could find the note I later sent to someone
>>>     or to some list, but I can't find it - sry).
>>>>>>>
>>>>>>>     I'd like some citations on this versus anecdata if possible
>>>>>>>     before I add that caveat to the text. Do large deployments
>>>>>>>     disable this? I haven't come across this suggestion in any
>>>>>>>     Linux tuning guides to date.
>>
>>     -- ________________________________________________________________
>>     Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/>
>>
>>     _______________________________________________
>>     tcpm mailing list
>>     tcpm@ietf.org <mailto:tcpm@ietf.org>
>>     https://www.ietf.org/mailman/listinfo/tcpm
>>     <https://www.ietf.org/mailman/listinfo/tcpm>
>>
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm