Re: [tcpm] [EXTERNAL] Re: Hystart and delay jitter

Praveen Balasubramanian <pravb@microsoft.com> Thu, 18 March 2021 22:37 UTC

Return-Path: <pravb@microsoft.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 883C73A0C86 for <tcpm@ietfa.amsl.com>; Thu, 18 Mar 2021 15:37:26 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.349
X-Spam-Level:
X-Spam-Status: No, score=-2.349 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.248, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6HrVPWl1fQRP for <tcpm@ietfa.amsl.com>; Thu, 18 Mar 2021 15:37:24 -0700 (PDT)
Received: from NAM06-BL2-obe.outbound.protection.outlook.com (mail-eopbgr650139.outbound.protection.outlook.com [40.107.65.139]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9BD923A0C84 for <tcpm@ietf.org>; Thu, 18 Mar 2021 15:37:24 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lBxEULGHCeFngC6JbsAbm/sNrU40FSd+ggZyAYvdt7N7cMtD6l1pxfZxObPOctYwrWlABcOIUW8Nmq2OCqS8BxUGGaTYhmrQDMclVOHukSJ/3Bf8oTGZ0p04f01z/9RNeuTvuLejDoGQG7pjaocdd0nz4LdVOdcTNYWl3atkIJakO7Sdyw6gvJ6mM6NhK++T1tA5SqKsOLq7V3kHPA2EmhXec9gdvAYDh9x1vmKVpquYhTOMjg03qdkruUbtncTYlZ9YsKxYvjFVUeoMaHQcXdi8GA0LCmZRQTJ6+LiyHjibnxBnbIsJrvk8b4m134i0RQl/h4Jff6NmHfY7R80fsQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lh5P1szGxvLuACuI2yIHtu06NPMXQJy+PcBULucin6s=; b=MdexaCIRM2Ef3NJtLRPsRX+N5exZe8NH/hUeJe+bXxA/S7/VgTt6+GxmAfuEiHnuagoq9LAwl5O48lkYYHWkgbMhrh/YVke6KeQoVMN6UK615tEjTCOkxBqkutAVcwu0bjoBtSxVvmPrSUwGUoyQhn1cAyBI3fCRhl8IdfPQiMaKoTMaWFScVT53CBilLaLK8c0cW2a/49B1z+GDmb6Nqg4INvddxhJNhc84stwcDiD9IBa0s5Zde7m211rtrIDkMjeg/h1nJMmB2luN78N70NMp3n4wtrGQiRvUDBePxnwhR44FYo1gVWw6h7HJk93UWzW6cZ+/Asi2+/RRFzTsHA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lh5P1szGxvLuACuI2yIHtu06NPMXQJy+PcBULucin6s=; b=AM8rdSdWTZ3CftnO1yrw+181KZVJKaXwWYTkvtN3TYVGJ293VewSpOQ0PlfkuAtWyypJZsyiw07k518ibqwZaQS3oXW2lRNT5LDtof4fK7dudFJht9yT6CJwg1+WLl+S3iNJbsj1qOysbHz+fF6EQS9lQkwaKfFzRAS4j2uU/M8=
Received: from CY1PR00MB0170.namprd00.prod.outlook.com (2a01:111:e400:c638::26) by CY1PR00MB0201.namprd00.prod.outlook.com (2a01:111:e400:c639::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3990.0; Thu, 18 Mar 2021 22:37:18 +0000
Received: from CY1PR00MB0170.namprd00.prod.outlook.com ([fe80::608f:899d:2294:e9f5]) by CY1PR00MB0170.namprd00.prod.outlook.com ([fe80::608f:899d:2294:e9f5%8]) with mapi id 15.20.4000.000; Thu, 18 Mar 2021 22:37:17 +0000
From: Praveen Balasubramanian <pravb@microsoft.com>
To: "ncardwell=40google.com@dmarc.ietf.org" <ncardwell=40google.com@dmarc.ietf.org>, Christian Huitema <huitema@huitema.net>
CC: "tcpm@ietf.org" <tcpm@ietf.org>, Yi Huang <huanyi@microsoft.com>, Matt Olson <maolson@microsoft.com>
Thread-Topic: [EXTERNAL] Re: [tcpm] Hystart and delay jitter
Thread-Index: AQHXF43Ijixf/iHizESn/C9J9cynIqqKXUpg
Date: Thu, 18 Mar 2021 22:37:17 +0000
Message-ID: <CY1PR00MB01700C2168E1B4200BE23B34B6699@CY1PR00MB0170.namprd00.prod.outlook.com>
References: <376bdc9f-4774-bfc8-1736-6c94fb24953c@huitema.net> <CADVnQymN6UH+XTgdkwdX16TsDTeeTu+S-=O1nVjQFWYbpDT24Q@mail.gmail.com>
In-Reply-To: <CADVnQymN6UH+XTgdkwdX16TsDTeeTu+S-=O1nVjQFWYbpDT24Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=09b6e272-b5ca-4c22-a421-e5ad318ef3b2; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-03-18T22:29:46Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
authentication-results: dmarc.ietf.org; dkim=none (message not signed) header.d=none;dmarc.ietf.org; dmarc=none action=none header.from=microsoft.com;
x-originating-ip: [2001:4898:80e8:3:95d5:3f73:f682:d302]
x-ms-publictraffictype: Email
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: 5a091fdd-063a-462e-ef68-08d8ea5e62e1
x-ms-traffictypediagnostic: CY1PR00MB0201:
x-ms-exchange-transport-forked: True
x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr
x-microsoft-antispam-prvs: <CY1PR00MB020178E13392E7ACDEB3020BB6699@CY1PR00MB0201.namprd00.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: mNOP8zCwKCdL2q8qcdl4clcHFAlsudBUw7uBnHux1ro8zEzixF9IxcclHzC4t8Y4vPQPvU+X3JUWs2zb1io4i7lJq3lqSOWazXvhv5Pg4QD1rGGw0qFl9OhVbUsMAffdGpceaKpFKdD9CrHUkztBtGKKSA7i9XpqEybypjmymnY/F8HwbLSuyAgiA+A7hEH1jh5cfpa+QF5tn1jynP3Oey5QSxgiKbtO5KniQ7zDEF/sMEsA7q9JQvdQUHFGd0B4oG0gHzIxVvulFNipmIWhTbv2vbquUXQMTy2V177i/1BM5oLaUD72aNaCfIA/PY7KTVQ7kDcXRMSYTGSXN8ScUSV0xQZa9l5UrR8M2hRZgflTrLHr7pnNg8kbpNwgHSDApFmoeIHPqPnfG7UDK2wheFfQB84qT42lvPu2xw+NDeoyNyn3snBhmqsy5jbKtKxgtSyVi3W/mCNN1sk8qZo10WZdSP7Yd0+TX+yCwLXF1sHnTphSqpE4QWwV/ou9maQqP9LDSQ2PqTPvVt0ME8uJEMIy07pif5Lh1lBfO38Fp5fqUV4m78JlYG3ULkE0hOqaX5/K/LsIxP/AY3OXdzy+eHhoaF4H1xEEjSemeaFYQbUkyeItOsUtO/XNZk/E+VaXN5gD8UhwBr2KRzj3YEg+essF60h+b/LnZkH5uwhRyxihIOyjb5zEvKkI0TForbMgN9x2YSIi2R2wCjBxCRpe5A==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CY1PR00MB0170.namprd00.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(376002)(366004)(396003)(346002)(136003)(5660300002)(9686003)(107886003)(66476007)(186003)(2906002)(478600001)(71200400001)(966005)(10290500003)(55016002)(76116006)(66556008)(110136005)(54906003)(53546011)(6506007)(66446008)(7696005)(8676002)(8990500004)(66946007)(8936002)(38100700001)(52536014)(316002)(83380400001)(86362001)(82950400001)(33656002)(82960400001)(4326008)(64756008); DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata: =?us-ascii?Q?34ehOv+IOb/Oo2P4c437NHxrHnSa3XxBv3WbbOhXbIQspM5jG6YSEIKnpt5R?= =?us-ascii?Q?ALljzdEO6zcCfayiYZ+zhZkrpW6+kl35JDc9Sw+t81ZyZGyS71OYCfvyUfcU?= =?us-ascii?Q?pUoGdsc6kXHomU868ZLFURL7k3p5w6XF5V4bOMOzdwxInfpgI0ienLU1GlCX?= =?us-ascii?Q?qDqMB3cryrF9zQPFmgNmGzq3LI4LbO7G7xDYp4mmd3NkxNu+nimnDhfiyQn0?= =?us-ascii?Q?lCd5qCskLKptz3/DeJ6vyAx147Gl349MOqEyhai2eXYkQwP6/pbxhiX2tO2+?= =?us-ascii?Q?86dOaqewFvqJMkax6J9XSsYHJQRg9HaLfYt3MgnEh2Vhq9Q++kpj9K8bcaYJ?= =?us-ascii?Q?/MYp73HJ5/pJlEKv9Mgrfwd/1ImLxX0G2OaKK5N5K2KCDe9h5YBYCW+sADIP?= =?us-ascii?Q?z2R0ufZDhJNzn6SXDtAOQkpc+GxQQ6x7+Uvky6pfDDqYFaV7e7QPrGRfiKr8?= =?us-ascii?Q?Ri0P4ixHqdzLqzkGHpPN71Dlc/kdkK5DEU0mnaJJtfuLzcHcfSSE2YeSObbP?= =?us-ascii?Q?SnMwNoBG+igKRcAniTQ95bpY8XtdUSltmHKah4FTJeFOl7nSxn3Su8KaPqTn?= =?us-ascii?Q?X/TDDIUu6pQooYqBR0ojUPqPQx3N1e9jrHBNH/8GrnH3xuxOC5yo6AQzOxKH?= =?us-ascii?Q?R2oyG0w+agvgBS+/YZiGS1U1X8dxGXN/Tu/QTrGKamyIbdnnit3OHFc/ASWF?= =?us-ascii?Q?vbrEfKfTpXpcTa8LmDB9CjIK3XMrkIrjTXGw8V8FSEUbSTqDbgValYYjHaZA?= =?us-ascii?Q?B+5kdFwjQ8oNo/DkyaEqHSQxtNTv/UTauRtltD4AjMghOXr+Z0guq4fWr8/W?= =?us-ascii?Q?xMTf94g4SMz2r46vfdcHiUGANPRkYMLHse3cD+85fdqMY9AhyjLDAS5Hp893?= =?us-ascii?Q?TLrHZH9ePlltof8lJz4DlRbyv/A8ybvXnEukzBmnwHkeJafFsIyQd1EIULrZ?= =?us-ascii?Q?fQx1/l903Dmv0816Vc7vvhlfxorzVBTOiF4AdfW63lRxfP4Rw7gu9iXgE6lb?= =?us-ascii?Q?hD3VNIjhZdNFklgR34alw9IWM+0f60xvod+fuZoN2jCbHP4nYgwRd7svS6O+?= =?us-ascii?Q?eBkRAWraMexHMnf06FtVRj/rKljC1hpUhjdHl9Seg7CTdAASupkdCGrkMDGv?= =?us-ascii?Q?+H98mOL6rAK7J8x9Gx2i4R2Lx31mJakyzeM7RhOf+qkrTblXQo2LfDcbvi8E?= =?us-ascii?Q?10V5/3Gsd0OgNTtsGF9qVIZ035hfaKrmxwCzQZS8VeyvHWcrNX32RrvgFcMg?= =?us-ascii?Q?+cS6sqblKe6XmiDr5YE0WVPLg/oAFrougue8gtafxX56tRtkRmyrE4X3YUFd?= =?us-ascii?Q?KSvcg7JnmXN1L8lfev1SO4nojMyNIyXw5dRk0J9Yeg+qhSIpUj4E/BxnGSTk?= =?us-ascii?Q?vYgnOmVyj9E2IOXN9GwDtTv2gt1T?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CY1PR00MB0170.namprd00.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5a091fdd-063a-462e-ef68-08d8ea5e62e1
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2021 22:37:17.8005 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 1ZLmBPIlDppgOMcutxiYAt4dj0qvw1UAjt2NPwfmhlF7YL3lRqh+8zIX5vmNeUTTYgN15R6+8jZnYjljHeka9QVhPrwAnx+lXv5NTaLlNPE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR00MB0201
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/ysOTz0-iDHcHh46F7SOetHkkwQc>
Subject: Re: [tcpm] [EXTERNAL] Re: Hystart and delay jitter
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Mar 2021 22:37:27 -0000

Thanks Christian for the suggestions! We will include a discussion on jitter in the next draft update and also experiment with the min of max function. 

-----Original Message-----
From: tcpm <tcpm-bounces@ietf.org> On Behalf Of Neal Cardwell
Sent: Friday, March 12, 2021 2:18 PM
To: Christian Huitema <huitema@huitema.net>
Cc: tcpm@ietf.org Extensions <tcpm@ietf.org>
Subject: [EXTERNAL] Re: [tcpm] Hystart and delay jitter

On Fri, Mar 12, 2021 at 4:29 PM Christian Huitema <huitema@huitema.net> wrote:
>
> Back in November 2019, when adding Cubic and Hystart to my 
> implementation of QUIC, I noticed that Hystart was sensitive to delay 
> jitter. Hystart detects the buildup of queues by monitoring the RTT.
> Some links experience delay jitter, caused for example by access 
> protocol for shared radio links or possibly by link-local ARQ protocols.
> The delay jitter can cause Hystart to make the wrong decision, in two ways:
>
> 1) Delay jitter during a previous period could cause some packets to 
> be delivered "faster than usual", causing Hystart to under-estimate 
> the min RTT for that period.
>
> 2) Delay jitter during the currently measured period can cause packets 
> to be delivered "slower than usual",  causing Hystart to over-estimate 
> the min RTT for that period.
>
> The combination of these two issues may cause Hystart to make the 
> wrong decisions, and exit slow start at levels well below link capacity.

Yes, we have found in both our production experience and controlled experiments that the Hystart-Delay algorithm is very susceptible to spurious triggering from jitter, particularly in LTE and wifi paths.

We discussed this a bit in Spring 2017 in the comparison of BBR's bandwidth-based mechanism for exiting startup, vs Hystart-Delay's delay-based mechanism:
  https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fproceedings%2F98%2Fslides%2Fslides-98-iccrg-an-update-on-bbr-congestion-control-00.pdf%23page%3D8&amp;data=04%7C01%7Cpravb%40microsoft.com%7Cbc3399870c0d4105299408d8e5a4cbaa%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637511843745826011%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=6SQWDb1mFH0cgrayEwiRYeccYvP0dv%2BAJ1eokyJF%2BKs%3D&amp;reserved=0

> The draft-ietf-tcpm-hystartplusplus-01 does have some protection 
> against the second issue, because currentRoundMinRTT is computed on at 
> least N_RTT_SAMPLE. If that number is large enough, computing the min 
> over N samples should filter out "slower than usual" anomalies. 
> However, the draft does not include a protection against "faster than usual"
> anomalies happening in the previous period. In my implementation, I 
> protected against that by computing a "min of max" function: compute a 
> rolling "MAX over N_RTT_SAMPLE", then compute the MIN value of that 
> during the reference period, and use that to set the reference value 
> "lastRoundMinRTT".
>
> I think it would be good to add a discussion of the effect of jitter 
> to the hystart++ draft. In addition, we may also want to mention 
> timestamps. The jitter on RTT may be caused by jitter on either 
> direction of transmission -- data path or ACK path. The effect of 
> jitter on the ACK path can be minimized if time stamps can be used to 
> monitor the variation of one-way delays. This is not discussed in the 
> current draft. Maybe it should be.

A discussion of the effect of jitter sounds like a great idea.

neal

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=04%7C01%7Cpravb%40microsoft.com%7Cbc3399870c0d4105299408d8e5a4cbaa%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637511843745826011%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=oqte6wCaTBUvrwgXVRMcD13YgtspMJyIWMMLlwNuYdY%3D&amp;reserved=0