Re: [tsvwg] The state of l4s, bbrv2, sce?
Dave Taht <dave.taht@gmail.com> Fri, 26 July 2019 15:45 UTC
Return-Path: <dave.taht@gmail.com>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 14C39120019 for <tsvwg@ietfa.amsl.com>; Fri, 26 Jul 2019 08:45:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.998
X-Spam-Level:
X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1WsdecTAHaQ9 for <tsvwg@ietfa.amsl.com>; Fri, 26 Jul 2019 08:45:44 -0700 (PDT)
Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E0D90120044 for <tsvwg@ietf.org>; Fri, 26 Jul 2019 08:45:43 -0700 (PDT)
Received: by mail-io1-xd2c.google.com with SMTP id h6so19152291iom.7 for <tsvwg@ietf.org>; Fri, 26 Jul 2019 08:45:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=TNxGzdEQv+DFS+gztgDOowiIe7MJXwpAobWfMgAs/SY=; b=a2uHIHKue5959pSDcsNnNOTgB15dnOfjvaKJ/xHqC83ZueMXTIxxx8co/qxhjt9EAW ASQja4KXFW2H6llFxIIb0b1nfhNmjqWtnUzB3SFExPjQ9bwqPV0tE7Jl4VU6JOnpL2ml FNUrxOgcY6tfJeSZxqwiUXI5W9lxTrMJ4wugy6WMYIMYtnS3k2jTBtZLV/tJj9OXaDVl AlRUHXAV3WYj0i3IxsRN/ziQhrGdkTQw3TfxQE1le0n3I4vZTG71xJ12fuAiBaCqbx00 sIIqX8OBqc2JSem6O+KcBy/yuKLUSqhVu7g2lHfct8vEkNLJh1OdysDyJqYA50d4UjMs st+w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=TNxGzdEQv+DFS+gztgDOowiIe7MJXwpAobWfMgAs/SY=; b=DqWxjdeYxO9ZqATwWXK0kqYEPOYnDe58d72nlDAxk3RwPesC6cj2iSpZFEXyXgK+2G QZ+z8gPdobGpw5ZVflLbR7ANNPgG7MGao4T7w8uwhYZRVHC6PJtpmkejhdZnHcuQYPCl iJ5HELI7YcO3lLPHpgLA3MhfK0E099BFoJhzl9Gi0siP6yNuWcHyJCMeAJ1VtSFxMR+X 8sW+/WE/gGgjklklwX0FvWe4KntnuRdCqejWnnZTm1oDtMzqMTc180K1Tz80IgALlG/D vBcZLQcUJEdxCCRyhjF7wMSG/wlnwRqsLybs0KFB9xsFNZpL0du4pMgxAK6jpxUJXrXW DPnA==
X-Gm-Message-State: APjAAAUXT2BNdb8vfkySBRFCQLyd3ZeqlvrxoIdmkDMD4mEfRAkRpsb+ X+EH7fruGD6lgwhvPESvit6GFOfDWrYRd0lW7N0=
X-Google-Smtp-Source: APXvYqwfHG/+0W/JvBBSice6ethSviXRUVigKwNLRrJ2p0Tr9LDdEmBWifI2fw6emYw784SvL6zm+//G8VBMebG2AMA=
X-Received: by 2002:a6b:dd17:: with SMTP id f23mr74480413ioc.213.1564155943157; Fri, 26 Jul 2019 08:45:43 -0700 (PDT)
MIME-Version: 1.0
References: <364514D5-07F2-4388-A2CD-35ED1AE38405@akamai.com> <1238A446-6E05-4A55-8B3B-878C8F39FC75@gmail.com> <AM4PR07MB3459B1173917DAFBCEB25511B9FA0@AM4PR07MB3459.eurprd07.prod.outlook.com> <17B33B39-D25A-432C-9037-3A4835CCC0E1@gmail.com> <AM4PR07MB345956F52D92759F24FFAA13B9F50@AM4PR07MB3459.eurprd07.prod.outlook.com> <52F85CFC-B7CF-4C7A-88B8-AE0879B3CCFE@gmail.com> <AM4PR07MB3459B471C4D7ADAE4CF713F3B9F60@AM4PR07MB3459.eurprd07.prod.outlook.com> <D231681B-1E57-44E1-992A-E8CC423926B6@akamai.com> <AM4PR07MB34592A10E2625C2C32B9893EB9F00@AM4PR07MB3459.eurprd07.prod.outlook.com> <A6F05DD3-D276-4893-9B15-F48E3018A129@gmx.de> <AM4PR07MB3459487C8A79B1152E132CE1B9CB0@AM4PR07MB3459.eurprd07.prod.outlook.com> <87ef2myqzv.fsf@taht.net> <a85d38ba-98ac-e43e-7610-658f4d03e0f4@mti-systems.com> <CE03DB3D7B45C245BCA0D243277949363062879C@MX307CL04.corp.emc.com> <803D9CA8-220E-4F98-9B8E-6CE2916C3100@gmail.com> <0079BC6B-4792-48ED-90D3-D9A69407F316@gmx.de> <22af0671-fdd0-0953-fc96-55b34beb0be9@bobbriscoe.net> <AC3C0A74-43C7-4351-B4FA-33AD2066B479@gmail.com> <VI1PR07MB34703F2998D9EE7B79BD4E60B9C40@VI1PR07MB3470.eurprd07.prod.outlook.com> <3EB0D59D-69A7-4730-BCDF-10E5C61EF987@heistp.net> <AM4PR07MB3459D891047B874E5AABB7D1B9C10@AM4PR07MB3459.eurprd07.prod.outlook.com> <B356F262-6047-4CA1-A6B8-B780C7981F63@heistp.net> <CAA93jw53+Cmt=c4ie9PZbOvq6fewor_nsQ65OuP1POWjmGErmg@mail.gmail.com> <CADVnQyn6H_6KnPZCw_4tQcRUopwfo3cpXvMdCLnF-y7RDtAFSQ@mail.gmail.com>
In-Reply-To: <CADVnQyn6H_6KnPZCw_4tQcRUopwfo3cpXvMdCLnF-y7RDtAFSQ@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
Date: Fri, 26 Jul 2019 08:45:30 -0700
Message-ID: <CAA93jw4V9oOEwUbz+oca3nSZmJ3uryiKEkmFz7xOwMy19ztMiw@mail.gmail.com>
To: Neal Cardwell <ncardwell@google.com>
Cc: Pete Heist <pete@heistp.net>, "De Schepper, Koen (Nokia - BE/Antwerp)" <koen.de_schepper@nokia-bell-labs.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, "tsvwg@ietf.org" <tsvwg@ietf.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/A9P2X7m_GVPddqUxdWUDK2UyDNw>
Subject: Re: [tsvwg] The state of l4s, bbrv2, sce?
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Jul 2019 15:45:46 -0000
On Fri, Jul 26, 2019 at 8:37 AM Neal Cardwell <ncardwell@google.com> wrote: > > On Fri, Jul 26, 2019 at 11:05 AM Dave Taht <dave.taht@gmail.com> wrote: >> >> 1) BBRv2 is now available for public hacking. I had a good readthrough >> last night. >> >> The published tree applies cleanly (with a small patch) to net-next. >> I've had a chance to read through the code (lots of good changes to >> bbr!). >> >> Although neal was careful to say in iccrg the optional ecn mode uses >> "dctcp/l4s-style signalling", he did not identify how that was >> actually applied >> at the middleboxes, and the supplied test scripts >> (gtests/net/tcp/bbr/nsperf) don't do that. All we know is that it's >> set to kick in at 20 packets. Is it fq_codel's ce_threshold? red? pie? >> dualpi? Does it revert to drop on overload? > > > As mentioned in the ICCRG session, the TCP source tree includes the scripts used to run the tests and generate the graphs in the slide deck. Here is the commit I was mentioning: > > https://github.com/google/bbr/commit/e76d4f89b0c42d5409d34c48ee6f8d32407d4b8d > > So you can look at exactly how each test was run, and re-run those tests yourself, with the v2alpha code or any experimental tweaks you might make beyond that. > > To answer your particular question, the ECN marks were from a bottleneck qdisc configured as: > > codel ce_threshold 242us limit 1000 target 100ms thx neal! I missed that! > I'm not claiming that's necessarily the best mechanism or set of parameters to set ECN marks. The 20-packet number comes from the DCTCP SIGCOMM 2010 paper's recommendation for 1Gbps bottlenecks. I just picked this kind of approach because the bare metal router/switch hardware varies, so this is a simple and easy way for everyone to experiment with the exact same ECN settings. ok! > >> Is it running on bare metal? 260us is at the bare bottom of what linux >> can schedule reliably, vms are much worse. > > > I have tried both VMs and bare metal with those scripts, and of course the VMs are quite noisy and the bare metal results much less noisy. So the graphs are from runs on bare metal x86 server-class machines. Good to know. On the cloud I use (linode) 1ms was the best I could hope for, and even then, dang jittery. (it was much worse 8 years back when xen underneath could be in the 10-20ms range!). There are major jitter issues on lower end hardware but I don't know how bad they are post spectre fixes, been afraid to look. containers are a huge improvement over vms but still break things like tsq. > > neal > -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
- [tsvwg] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] Comments on L4S drafts Ingemar Johansson S
- Re: [tsvwg] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] Comments on L4S drafts Luca Muscariello
- Re: [tsvwg] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] Comments on L4S drafts Ingemar Johansson S
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Ingemar Johansson S
- Re: [tsvwg] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] Comments on L4S drafts Luca Muscariello
- Re: [tsvwg] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Luca Muscariello
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Holland, Jake
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Gorry Fairhurst
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Wesley Eddy
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Black, David
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Wesley Eddy
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Scharf, Michael
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Gorry Fairhurst
- [tsvwg] Hackathon tests Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Black, David
- Re: [tsvwg] New Version Notification for draft-gr… alex.burr@ealdwulf.org.uk
- Re: [tsvwg] New Version Notification for draft-gr… Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] New Version Notification for draft-gr… alex.burr@ealdwulf.org.uk
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [tcpm] New Version Notification for d… Scharf, Michael
- Re: [tsvwg] [tcpm] New Version Notification for d… Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bless, Roland (TM)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Pete Heist
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Dave Taht
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts De Schepper, Koen (Nokia - BE/Antwerp)
- Re: [tsvwg] [tcpm] New Version Notification for d… Bob Briscoe
- Re: [tsvwg] [tcpm] New Version Notification for d… Scharf, Michael
- Re: [tsvwg] [tcpm] New Version Notification for d… Black, David
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Bob Briscoe
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Sebastian Moeller
- [tsvwg] [Ecn-sane] Compatibility with singlw queu… Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Comments on L4S drafts Pete Heist
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Black, David
- [tsvwg] The state of l4s, bbrv2, sce? Dave Taht
- Re: [tsvwg] The state of l4s, bbrv2, sce? Dave Taht
- Re: [tsvwg] The state of l4s, bbrv2, sce? Neal Cardwell
- Re: [tsvwg] The state of l4s, bbrv2, sce? Dave Taht
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Holland, Jake
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Black, David
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Black, David
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Sebastian Moeller
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Jonathan Morton
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Mikael Abrahamsson
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … Mikael Abrahamsson
- Re: [tsvwg] [Ecn-sane] Compatibility with singlw … John Leslie
- Re: [tsvwg] New Version Notification for draft-gr… Rodney W. Grimes
- Re: [tsvwg] New Version Notification for draft-gr… Gorry Fairhurst
- Re: [tsvwg] New Version Notification for draft-gr… Jonathan Morton
- Re: [tsvwg] New Version Notification for draft-gr… Gorry Fairhurst
- Re: [tsvwg] New Version Notification for draft-gr… Sebastian Moeller
- Re: [tsvwg] New Version Notification for draft-gr… Gorry Fairhurst
- Re: [tsvwg] [tcpm] New Version Notification for d… Scharf, Michael
- Re: [tsvwg] New Version Notification for draft-gr… Rodney W. Grimes
- Re: [tsvwg] [tcpm] New Version Notification for d… Loganaden Velvindron
- Re: [tsvwg] [tcpm] New Version Notification for d… Black, David