Re: [tcpm] Hystart survey of large server operators

Bob Briscoe <ietf@bobbriscoe.net> Thu, 29 July 2021 16:28 UTC

Return-Path: <ietf@bobbriscoe.net>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B6273A0B01; Thu, 29 Jul 2021 09:28:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tvWIeqmgYI5m; Thu, 29 Jul 2021 09:28:20 -0700 (PDT)
Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 151B03A0B1A; Thu, 29 Jul 2021 09:28:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BPrGtN716r4yvWd+B0bESPrc5fXvSINzbyDMGuypfkY=; b=x+MyFkJvxmHRRw4HHu1g7Zcsho HvXzprvamGb3ZMzBWaSwb/bGaX/XW7ZiS9qadO8DmZbq7uPVtOCUiScag28QrhO/05KOLrR7DsZ9P Ua6qUH97YlMRePYPIbbQyz+dzoMwQlwzl5bajBDPcmnnv0ZQvfezKzyQcEo7r41l7vu15Aha2cZR0 ZRQrbbdpZ9kSSnCkWhh9+tGO9zeTshteTU7l3lP6yaYpmPHNQLmPGModiEbjI0yGvDQsL/lZoJOtL v17j9CF5x8CYJC+LNCFBnshf91GRJGilBbhDPZB3ioZdSihQym0vDKyHVUDI88Jc/cjensEW6+/82 VDDt4jrg==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:52882 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from <ietf@bobbriscoe.net>) id 1m98tT-0001cI-4a; Thu, 29 Jul 2021 17:28:16 +0100
To: Yuchung Cheng <ycheng@google.com>, huitema@huitema.net
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, "draft-ietf-tcpm-hystartplusplus@ietf.org" <draft-ietf-tcpm-hystartplusplus@ietf.org>, Neal Cardwell <ncardwell@google.com>
References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <PH0PR00MB1030126A3220BC056A406490B6E99@PH0PR00MB1030.namprd00.prod.outlook.com> <PH0PR00MB1030E0697BE8E93074B901D2B6E99@PH0PR00MB1030.namprd00.prod.outlook.com> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <CAK6E8=f7qKDs-MFr4G6bpz82Swn3iCJEkWLL5yr+vV8z9zMb=Q@mail.gmail.com> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> <CAK6E8=dpdFjcEoPkMORw+CjFAmpo+ZtS-Scj6qh9oYSmbtZyWA@mail.gmail.com>
From: Bob Briscoe <ietf@bobbriscoe.net>
Message-ID: <a746b460-cce8-c061-ae11-5773a70c2a0a@bobbriscoe.net>
Date: Thu, 29 Jul 2021 17:28:14 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <CAK6E8=dpdFjcEoPkMORw+CjFAmpo+ZtS-Scj6qh9oYSmbtZyWA@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------66B5964856B4360959048994"
Content-Language: en-GB
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net
X-Source:
X-Source-Args:
X-Source-Dir:
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/oLzDx1UX7MHYTlO77EmKwJEzwnQ>
Subject: Re: [tcpm] Hystart survey of large server operators
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 29 Jul 2021 16:28:26 -0000

Yuchung,

On 29/07/2021 00:35, Yuchung Cheng wrote:
> same experience with Chrisitian
>
> For more than half a decade Google traffic used Linux CUBIC Hystart. 
> This includes google.com <http://google.com/> and YouTube public 
> Internet traffic, as well as internal Google TCP traffic. When we 
> deployed pacing around 2013 we disabled the Hystart ACK-train 
> mechanism (that Hystart++ prohibits now) as it causes false SS 
> exit. But we continued to use the stock Hystart Delay mechanism. 
> ~2014-15 we switched to BBR which uses a different startup more robust 
> to delay jitters.

[BB] You or Neal (sry can't remember which) were one of those I surveyed 
who said they had disabled Hystart. So perhaps this was a 
misunderstanding over the ACK-train part.

> But IMO the simpler hystart mitigating SS over-shoot is still highly 
> valuable for Cubic and other congestion controls, as Internet 
> bandwidth continues to hike.
>
> Did those providers disable hystart b/c of its poor interaction 
> between the ack-train mode and pacing?

[BB] I didn't ask them why.



Bob

> Maybe it's time to disable the ack-train approach in the upstream 
> Linux when hystart++ is standardized.
>
> On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <huitema@huitema.net 
> <mailto:huitema@huitema.net>> wrote:
>
>     I certainly do not qualify as a "large operator", but when
>     implementing Cubic for QUIC I found that Hystart was critical for
>     performance. Specifically, the classical slow-start is very prone
>     to overshooting the capacity of the path, which causes large
>     batches of errors. With web-like traffic, these batches of errors
>     cause increase latency for the first transactions in a session,
>     and thus a drop in perceived quality. Hystart solves that. With
>     Hystart, I observed that a large fraction of sessions do not
>     experience any packet loss at all.
>
>     As an aside, we should strive for this suppression of packet
>     losses. That's actually a big reason for moving from Cubic to BBR.
>     Hystart suppresses the losses during the initial phase of the
>     connection, but Cubic still relies on periodically testing the
>     limit of path capacity and causing losses during the subsequent
>     phase. BBR's probing for bottleneck bandwidth is much more
>     conservative, does not cause such losses. It might be possible to
>     adapt Cubic to also not cause losses, for example by ending an
>     epoch early if too many CE marks are received or if the RTT
>     increases. That would be worth trying.
>
>     -- Christian Huitema
>
>     On 7/28/2021 3:19 AM, Bob Briscoe wrote:
>>     Yuchung,
>>
>>     It was during a Mar 2018 ad hoc workshop Jana had organized at
>>     the London IETF entitled 'BBR and the intersection with other
>>     work". I can't remember why I needed to know at the time, but
>>     during the break I approached the major server operators
>>     individually, established whether they used Cubic, and if so
>>     asked whether they used Hystart or disabled it. It's hard to
>>     anonymize the results, because IMMSMC all those that used Cubic
>>     said they disabled Hystart.
>>
>>     If this isn't correct, then maybe people who replied at the time
>>     thought they were being asked something else. Or maybe it was
>>     correct then but isn't now.
>>
>>
>>     Bob
>>
>>
>>     On 28/07/2021 00:46, Yuchung Cheng wrote:
>>>     Wait -- how is hystart "invariably disabled" in Linux (cubic)!?
>>>
>>>     What data indicates that
>>>
>>>     On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net
>>>     <mailto:ietf@bobbriscoe.net> <mailto:ietf@bobbriscoe.net>
>>>     <mailto:ietf@bobbriscoe.net>> wrote:
>>>
>>>         Any large server operator out there who are using Cubic,
>>>         If you're willing to state whether or not your operations
>>>     disable
>>>         Hystart, pls do so in reply.
>>>         Then Praveen can cite this mailing list thread in the Hystart++
>>>         draft, as requested below.
>>>
>>>         If you are willing to reply privately, I would be willing to
>>>     keep
>>>         your confidences, and provide an anonymized result to the list.
>>>
>>>         Cheers
>>>
>>>
>>>
>>>         Bob
>>>
>>>         On 27/07/2021 01:50, Praveen Balasubramanian wrote:
>>>>
>>>>         Although Hystart is default enabled in Linux, it is invariably
>>>>         disabled. So, it's misleading to just say Hystart is default
>>>>         enabled, which implies it's widely used, when people
>>>>     clearly find
>>>>         it has problems (which motivates Hystart++). I found this out
>>>>         through an informal survey I did at the Mar'18 IETF in
>>>>     London by
>>>>         asking round the implementers of the most prevalent stacks (I
>>>>         would name names if I could find the note I later sent to
>>>>     someone
>>>>         or to some list, but I can't find it - sry).
>>>>>>>>
>>>>>>>>         I'd like some citations on this versus anecdata if
>>>>>>>>     possible
>>>>>>>>         before I add that caveat to the text. Do large deployments
>>>>>>>>         disable this? I haven't come across this suggestion in any
>>>>>>>>         Linux tuning guides to date.
>>>
>>>         --
>>>     ________________________________________________________________
>>>         Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/>
>>>     <http://bobbriscoe.net/> <http://bobbriscoe.net/>
>>>
>>>         _______________________________________________
>>>         tcpm mailing list
>>>     tcpm@ietf.org <mailto:tcpm@ietf.org> <mailto:tcpm@ietf.org>
>>>     <mailto:tcpm@ietf.org>
>>>     https://www.ietf.org/mailman/listinfo/tcpm
>>>     <https://www.ietf.org/mailman/listinfo/tcpm>
>>>     <https://www.ietf.org/mailman/listinfo/tcpm>
>>>     <https://www.ietf.org/mailman/listinfo/tcpm>
>>>
>>
>>
>>     _______________________________________________
>>     tcpm mailing list
>>     tcpm@ietf.org  <mailto:tcpm@ietf.org>
>>     https://www.ietf.org/mailman/listinfo/tcpm  <https://www.ietf.org/mailman/listinfo/tcpm>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/