Re: [aqm] [Bloat] TCP BBR paper is now generally available

Neal Cardwell <> Sat, 03 December 2016 13:04 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id CF5DD1297A7 for <>; Sat, 3 Dec 2016 05:04:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -4.897
X-Spam-Status: No, score=-4.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RP_MATCHES_RCVD=-2.896, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id p05zls6hkrem for <>; Sat, 3 Dec 2016 05:04:22 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:4003:c06::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 30B611297A5 for <>; Sat, 3 Dec 2016 05:04:22 -0800 (PST)
Received: by with SMTP id w63so295515908oiw.0 for <>; Sat, 03 Dec 2016 05:04:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=lzksxY68T/ATjMu+hW2qq1QIl7AkAsJrUBMuHBgFSpw=; b=FF87ebDbos9Vy8/pylTPLCkaMKAYtVg6jFtFD9Hs28sJnmErfAz2p7ePNE9w7UmMiC YbpBiw/fkmRaASFx3tsIe+t7GkYTEkFGfavIgCytylIJgy5jUDiHGFNxD4366698GSfD tNrZtdNj1BCi9HRzlPBWgFfmRiuWk/LN/7UIEQNrpYWFccrOD8q35q+dkWcvKLcvsq0r Wmwy+hoXNRnlbq1+j3zrroQYgoaJWR8PqG/5G0NmNadRO89SVrWj8aIo1t5ImGms5zjJ 29fzCRLlrCZGhix4l7JAA9m+GA7UdgrW/xIl/CqGo4PAh5pwGU5Peanq6extO8q66bjH 9ieA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=lzksxY68T/ATjMu+hW2qq1QIl7AkAsJrUBMuHBgFSpw=; b=VnsJ/71EhSIgmgilWQ/IXSKXgsvdQdpU3Kz08TqLKky2NZUPpPn1GLn/XY1BLYL8UZ zhBA9I7HxEvQ0ouevNmKTwoGQDOEfIawCZh6fXcg0HVMr0kK4TSSm5CKsfFCJMR0JdC/ XVMM7BuDb7kv4OLt+Bd+HrIRAgvbNTcMUe2pIszYAI9f1mi8wXvGVOuj2sFYc2wP0wvb nSixmP5vG10KH7CAGxQjtiEmmZhay3WmgAyMzJAydGGVya0cm34Y9uucuZ68W6EYyXkB GAfgxlAMH6FyjgceQDKEt7F/NJ6xMj55zFofSIy28uZhYHCN9+zMQc7Ekwnv6zXRts9u ERlA==
X-Gm-Message-State: AKaTC02wm6cEJjigLMOjdO6wXQTBOiHtq6ZXgIS37mOCXVrrJAtPPU4HOXdINSltKhYmCxEnyaMZ/bHanBud3tAX
X-Received: by with SMTP id n81mr24432133oih.94.1480770261345; Sat, 03 Dec 2016 05:04:21 -0800 (PST)
MIME-Version: 1.0
Received: by with HTTP; Sat, 3 Dec 2016 05:03:50 -0800 (PST)
In-Reply-To: <>
References: <> <> <> <> <> <>
From: Neal Cardwell <>
Date: Sat, 3 Dec 2016 08:03:50 -0500
Message-ID: <>
To: Eric Dumazet <>
Content-Type: text/plain; charset=UTF-8
Archived-At: <>
Cc: "Steinar H. Gunderson" <>, bloat <>, "" <>, Jonathan Morton <>
Subject: Re: [aqm] [Bloat] TCP BBR paper is now generally available
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sat, 03 Dec 2016 13:04:24 -0000

Thanks for the report, Steinar. This is the first report we've had
like this, but it would be interesting to find out what's going on.

Even if you don't have time to apply the patches Eric mentions, it
would be hugely useful if the next time you have a slow transfer like
that you could post a link to a tcpdump packet capture (headers only
is best, say -s 120). Ideally the trace would capture a whole
connection, so we can see the wscale on the SYN exchange.


On Fri, Dec 2, 2016 at 6:31 PM, Eric Dumazet <> wrote:
> On Fri, 2016-12-02 at 23:40 +0100, Steinar H. Gunderson wrote:
>> On Fri, Dec 02, 2016 at 05:22:23PM -0500, Neal Cardwell wrote:
>> > Of course, if we find important use cases that don't work with BBR, we will
>> > see what we can do to make BBR work well with them.
>> I have one thing that I _wonder_ if could be BBR's fault: I run backup over
>> SSH. (That would be tar + gzip + ssh.) The first full backup after I rolled
>> out BBR on the server (the one sending the data) suddenly was very slow
>> (~50 Mbit/sec); there was plenty of free I/O, and neither tar nor gzip
>> (well, pigz) used a full core. My only remaining explanation would be that
>> somehow, BBR didn't deal well with the irregular stream of data coming from
>> tar. (A wget between the same machines at the same time gave 6-700 Mbit/sec.)
>> I will not really blame BBR here, since I didn't take a tcpdump or have time
>> to otherwise debug properly (short of eliminating the other things I already
>> mentioned); most likely, it's something else. But if you've ever heard of
>> others with similar issues, consider this a second report. :-)
>> /* Steinar */
> It would be interesting to get the chrono stats for the TCP flow, with
> an updated ss/iproute2 command and the kernel patches :
> efd90174167530c67a54273fd5d8369c87f9bd32 tcp: export sender limits chronographs to TCP_INFO
> b0f71bd3e190df827d25d7f19bf09037567f14b7 tcp: instrument how long TCP is limited by insufficient send buffer
> 5615f88614a47d2b802e1d14d31b623696109276 tcp: instrument how long TCP is limited by receive window
> 0f87230d1a6c253681550c6064715d06a32be73d tcp: instrument how long TCP is busy sending
> 05b055e89121394058c75dc354e9a46e1e765579 tcp: instrument tcp sender limits chronographs