Re: [tcpm] Hystart survey of large server operators
Bob Briscoe <ietf@bobbriscoe.net> Thu, 29 July 2021 16:28 UTC
Return-Path: <ietf@bobbriscoe.net>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B6273A0B01; Thu, 29 Jul 2021 09:28:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tvWIeqmgYI5m; Thu, 29 Jul 2021 09:28:20 -0700 (PDT)
Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 151B03A0B1A; Thu, 29 Jul 2021 09:28:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BPrGtN716r4yvWd+B0bESPrc5fXvSINzbyDMGuypfkY=; b=x+MyFkJvxmHRRw4HHu1g7Zcsho HvXzprvamGb3ZMzBWaSwb/bGaX/XW7ZiS9qadO8DmZbq7uPVtOCUiScag28QrhO/05KOLrR7DsZ9P Ua6qUH97YlMRePYPIbbQyz+dzoMwQlwzl5bajBDPcmnnv0ZQvfezKzyQcEo7r41l7vu15Aha2cZR0 ZRQrbbdpZ9kSSnCkWhh9+tGO9zeTshteTU7l3lP6yaYpmPHNQLmPGModiEbjI0yGvDQsL/lZoJOtL v17j9CF5x8CYJC+LNCFBnshf91GRJGilBbhDPZB3ioZdSihQym0vDKyHVUDI88Jc/cjensEW6+/82 VDDt4jrg==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:52882 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from <ietf@bobbriscoe.net>) id 1m98tT-0001cI-4a; Thu, 29 Jul 2021 17:28:16 +0100
To: Yuchung Cheng <ycheng@google.com>, huitema@huitema.net
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, "draft-ietf-tcpm-hystartplusplus@ietf.org" <draft-ietf-tcpm-hystartplusplus@ietf.org>, Neal Cardwell <ncardwell@google.com>
References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <PH0PR00MB1030126A3220BC056A406490B6E99@PH0PR00MB1030.namprd00.prod.outlook.com> <PH0PR00MB1030E0697BE8E93074B901D2B6E99@PH0PR00MB1030.namprd00.prod.outlook.com> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <CAK6E8=f7qKDs-MFr4G6bpz82Swn3iCJEkWLL5yr+vV8z9zMb=Q@mail.gmail.com> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> <CAK6E8=dpdFjcEoPkMORw+CjFAmpo+ZtS-Scj6qh9oYSmbtZyWA@mail.gmail.com>
From: Bob Briscoe <ietf@bobbriscoe.net>
Message-ID: <a746b460-cce8-c061-ae11-5773a70c2a0a@bobbriscoe.net>
Date: Thu, 29 Jul 2021 17:28:14 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <CAK6E8=dpdFjcEoPkMORw+CjFAmpo+ZtS-Scj6qh9oYSmbtZyWA@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------66B5964856B4360959048994"
Content-Language: en-GB
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net
X-Source:
X-Source-Args:
X-Source-Dir:
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/oLzDx1UX7MHYTlO77EmKwJEzwnQ>
Subject: Re: [tcpm] Hystart survey of large server operators
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 29 Jul 2021 16:28:26 -0000
Yuchung, On 29/07/2021 00:35, Yuchung Cheng wrote: > same experience with Chrisitian > > For more than half a decade Google traffic used Linux CUBIC Hystart. > This includes google.com <http://google.com/> and YouTube public > Internet traffic, as well as internal Google TCP traffic. When we > deployed pacing around 2013 we disabled the Hystart ACK-train > mechanism (that Hystart++ prohibits now) as it causes false SS > exit. But we continued to use the stock Hystart Delay mechanism. > ~2014-15 we switched to BBR which uses a different startup more robust > to delay jitters. [BB] You or Neal (sry can't remember which) were one of those I surveyed who said they had disabled Hystart. So perhaps this was a misunderstanding over the ACK-train part. > But IMO the simpler hystart mitigating SS over-shoot is still highly > valuable for Cubic and other congestion controls, as Internet > bandwidth continues to hike. > > Did those providers disable hystart b/c of its poor interaction > between the ack-train mode and pacing? [BB] I didn't ask them why. Bob > Maybe it's time to disable the ack-train approach in the upstream > Linux when hystart++ is standardized. > > On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <huitema@huitema.net > <mailto:huitema@huitema.net>> wrote: > > I certainly do not qualify as a "large operator", but when > implementing Cubic for QUIC I found that Hystart was critical for > performance. Specifically, the classical slow-start is very prone > to overshooting the capacity of the path, which causes large > batches of errors. With web-like traffic, these batches of errors > cause increase latency for the first transactions in a session, > and thus a drop in perceived quality. Hystart solves that. With > Hystart, I observed that a large fraction of sessions do not > experience any packet loss at all. > > As an aside, we should strive for this suppression of packet > losses. That's actually a big reason for moving from Cubic to BBR. > Hystart suppresses the losses during the initial phase of the > connection, but Cubic still relies on periodically testing the > limit of path capacity and causing losses during the subsequent > phase. BBR's probing for bottleneck bandwidth is much more > conservative, does not cause such losses. It might be possible to > adapt Cubic to also not cause losses, for example by ending an > epoch early if too many CE marks are received or if the RTT > increases. That would be worth trying. > > -- Christian Huitema > > On 7/28/2021 3:19 AM, Bob Briscoe wrote: >> Yuchung, >> >> It was during a Mar 2018 ad hoc workshop Jana had organized at >> the London IETF entitled 'BBR and the intersection with other >> work". I can't remember why I needed to know at the time, but >> during the break I approached the major server operators >> individually, established whether they used Cubic, and if so >> asked whether they used Hystart or disabled it. It's hard to >> anonymize the results, because IMMSMC all those that used Cubic >> said they disabled Hystart. >> >> If this isn't correct, then maybe people who replied at the time >> thought they were being asked something else. Or maybe it was >> correct then but isn't now. >> >> >> Bob >> >> >> On 28/07/2021 00:46, Yuchung Cheng wrote: >>> Wait -- how is hystart "invariably disabled" in Linux (cubic)!? >>> >>> What data indicates that >>> >>> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net >>> <mailto:ietf@bobbriscoe.net> <mailto:ietf@bobbriscoe.net> >>> <mailto:ietf@bobbriscoe.net>> wrote: >>> >>> Any large server operator out there who are using Cubic, >>> If you're willing to state whether or not your operations >>> disable >>> Hystart, pls do so in reply. >>> Then Praveen can cite this mailing list thread in the Hystart++ >>> draft, as requested below. >>> >>> If you are willing to reply privately, I would be willing to >>> keep >>> your confidences, and provide an anonymized result to the list. >>> >>> Cheers >>> >>> >>> >>> Bob >>> >>> On 27/07/2021 01:50, Praveen Balasubramanian wrote: >>>> >>>> Although Hystart is default enabled in Linux, it is invariably >>>> disabled. So, it's misleading to just say Hystart is default >>>> enabled, which implies it's widely used, when people >>>> clearly find >>>> it has problems (which motivates Hystart++). I found this out >>>> through an informal survey I did at the Mar'18 IETF in >>>> London by >>>> asking round the implementers of the most prevalent stacks (I >>>> would name names if I could find the note I later sent to >>>> someone >>>> or to some list, but I can't find it - sry). >>>>>>>> >>>>>>>> I'd like some citations on this versus anecdata if >>>>>>>> possible >>>>>>>> before I add that caveat to the text. Do large deployments >>>>>>>> disable this? I haven't come across this suggestion in any >>>>>>>> Linux tuning guides to date. >>> >>> -- >>> ________________________________________________________________ >>> Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/> >>> <http://bobbriscoe.net/> <http://bobbriscoe.net/> >>> >>> _______________________________________________ >>> tcpm mailing list >>> tcpm@ietf.org <mailto:tcpm@ietf.org> <mailto:tcpm@ietf.org> >>> <mailto:tcpm@ietf.org> >>> https://www.ietf.org/mailman/listinfo/tcpm >>> <https://www.ietf.org/mailman/listinfo/tcpm> >>> <https://www.ietf.org/mailman/listinfo/tcpm> >>> <https://www.ietf.org/mailman/listinfo/tcpm> >>> >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org <mailto:tcpm@ietf.org> >> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm> > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/
- [tcpm] I-D Action: draft-ietf-tcpm-hystartplusplu… internet-drafts
- [tcpm] Full review of: draft-ietf-tcpm-hystartplu… Bob Briscoe
- [tcpm] Full review of: draft-ietf-tcpm-hystartplu… Bob Briscoe
- Re: [tcpm] Full review of: draft-ietf-tcpm-hystar… Bob Briscoe
- Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-… Praveen Balasubramanian
- Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-… Praveen Balasubramanian
- Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-… Praveen Balasubramanian
- Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-… Praveen Balasubramanian
- [tcpm] Hystart survey of large server operators Bob Briscoe
- Re: [tcpm] Hystart survey of large server operato… Yuchung Cheng
- Re: [tcpm] Hystart survey of large server operato… Bob Briscoe
- Re: [tcpm] Hystart survey of large server operato… Christian Huitema
- Re: [tcpm] Hystart survey of large server operato… Yuchung Cheng
- Re: [tcpm] Hystart survey of large server operato… Bob Briscoe
- Re: [tcpm] Hystart survey of large server operato… Yuchung Cheng
- Re: [tcpm] Hystart survey of large server operato… Neal Cardwell
- Re: [tcpm] Hystart survey of large server operato… Junho Choi