Re: [tsvwg] The state of l4s, bbrv2, sce?

Neal Cardwell <> Fri, 26 July 2019 15:37 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id F2DA812015E for <>; Fri, 26 Jul 2019 08:37:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -17.5
X-Spam-Status: No, score=-17.5 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id m6OedFTVFOk1 for <>; Fri, 26 Jul 2019 08:37:31 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::32c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id D403F120161 for <>; Fri, 26 Jul 2019 08:37:31 -0700 (PDT)
Received: by with SMTP id b7so5610079otl.11 for <>; Fri, 26 Jul 2019 08:37:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=p2wHehMJrtRgYJYaa4/6XEq6BErL4+Kf/TDNsMxM+eA=; b=Z2TFOv66hZb6RyI2AmZH5EZwsW9IjfQra8VVyC97vMAsQgKH8qJ3ZXXxlYxfkjbjCm UaMxhu9msyoNDScGBrOo/ARX8vktGsJ4o3ANXd+Zl+wxCx/qBr3kpYslBh9/aZidjQDN bxnXpqcLdEbHAUfOHN38/TnM2gvbcFZ4xZ7tSijM5SbYLp9cPBB9wDQdzP4MTIItDAsg z5lm3zcRYjID4FVWHjfPsB9cWnaav4YuFzGpg10NCeIeNvmoRmobEb26dg0d5QYyeQ9i rWlydPn+XhHvr1aC5Te/XAQ+afc7HTNB0wReEuXzIGsdGq4rwfTr4RQDKCxQ5zYlTGMA ta1g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=p2wHehMJrtRgYJYaa4/6XEq6BErL4+Kf/TDNsMxM+eA=; b=udKWxG5SqqJI52b9dHnzcrBslJPUrIpoTD+HvnyY2M6cvN73m0V29HwCubbSFl0Azq NKKVISM381V8OyRAeTJ6XmzGL89hHqbFyGCpINfE4pBdV6TtC3bAhhYbK7ixnq49GTcw Djdd6jwiGFk8oDToe1yuA0layrcuWBizanW7FffHYXPUl1QshVbBzIS2f0CFPS+96xKU rJSkx40MoVLnjArqcJTsgbMM+tOvndChbcGcNGEVWwVpW2rS2QBLH39mcVHpdryWQ8VN 6n89MhfuSLMnJh3uqVQyLA4VpghY2AwnA4CfFTMRtVg7s4ZIt1rI3rCBFAiHfeLgvI0y Kt7g==
X-Gm-Message-State: APjAAAX1C7RSk60PtK97FtlErMPbSFqwagXcISGIEtd+zIND75mDd2O+ uhOI0PsBr8z9/T/sGNsToh6ulJ1yEJVp4KcYNXZwkQ==
X-Google-Smtp-Source: APXvYqyqtEcqmnZwH4E446zsyyPTr8Yx+2Qy0VY/CyzT6jvxeBelbwl+1XAnTdSSHh5qBWtYlEJ7Bl6uiV6BJIoyWsg=
X-Received: by 2002:a9d:5cc1:: with SMTP id r1mr64811747oti.341.1564155450666; Fri, 26 Jul 2019 08:37:30 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
From: Neal Cardwell <>
Date: Fri, 26 Jul 2019 11:37:13 -0400
Message-ID: <>
To: Dave Taht <>
Cc: Pete Heist <>, "De Schepper, Koen (Nokia - BE/Antwerp)" <>, "" <>, "" <>
Content-Type: multipart/alternative; boundary="000000000000033b37058e975229"
Archived-At: <>
Subject: Re: [tsvwg] The state of l4s, bbrv2, sce?
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 26 Jul 2019 15:37:34 -0000

On Fri, Jul 26, 2019 at 11:05 AM Dave Taht <> wrote:

> 1) BBRv2 is now available for public hacking. I had a good readthrough
> last night.
> The published tree applies cleanly (with a small patch) to net-next.
> I've had a chance to read through the code (lots of good changes to
> bbr!).
> Although neal was careful to say in iccrg the optional ecn mode uses
> "dctcp/l4s-style signalling", he did not identify how that was
> actually applied
> at the middleboxes, and the supplied test scripts
> (gtests/net/tcp/bbr/nsperf) don't do that. All we know is that it's
> set to kick in at 20 packets. Is it fq_codel's ce_threshold? red? pie?
> dualpi? Does it revert to drop on overload?

As mentioned in the ICCRG session, the TCP source tree includes the scripts
used to run the tests and generate the graphs in the slide deck. Here is
the commit I was mentioning:

So you can look at exactly how each test was run, and re-run those tests
yourself, with the v2alpha code or any experimental tweaks you might make
beyond that.

To answer your particular question, the ECN marks were from a bottleneck
qdisc configured as:

  codel ce_threshold 242us limit 1000 target 100ms

I'm not claiming that's necessarily the best mechanism or set of parameters
to set ECN marks. The 20-packet number comes from the DCTCP SIGCOMM 2010
paper's recommendation for 1Gbps bottlenecks. I just picked this kind of
approach because the bare metal router/switch hardware varies, so this is a
simple and easy way for everyone to experiment with the exact same ECN

Is it running on bare metal? 260us is at the bare bottom of what linux
> can schedule reliably, vms are much worse.

I have tried both VMs and bare metal with those scripts, and of course the
VMs are quite noisy and the bare metal results much less noisy. So the
graphs are from runs on bare metal x86 server-class machines.