Re: [aqm] [Bloat] TCP BBR paper is now generally available

Eric Dumazet <> Fri, 02 December 2016 23:32 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id BC5CC129478 for <>; Fri, 2 Dec 2016 15:32:30 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.7
X-Spam-Status: No, score=-2.7 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id I4K14S7dZI1m for <>; Fri, 2 Dec 2016 15:32:28 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:400e:c00::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 09E8012945F for <>; Fri, 2 Dec 2016 15:32:27 -0800 (PST)
Received: by with SMTP id 144so14193648pfv.0 for <>; Fri, 02 Dec 2016 15:32:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=6PdI3E9zpyhugi7nmQXDwciegDKofKFicESl7yyogU0=; b=rg5AROICO3f7rM9cizARv9MN0xp/KQS6A+WdCKZkTjH5ieaaeDnqzMsAOUSy8kP/CL 2l+f6l6C0VsdIxeCsgWnJuPau9JgzBxxO7gMFehA4St4elDppCvsKzZaXh954m1UZ0LB GYP+tM2kuIr9f+KIFfY45+xagu5iU0AsBe7mJB0m2+tRmUHrBogLoTjQ7HWsiom8Q4m/ INV8lYVSBUwJJgqG3iu9IOrjN2rpZlGFatoMmrg4/1K3LQXS2UxxgVeUN4wbrjeKcNqI pUMjmZ+CPRG9lGrx0GdDrv9/N7BThw8z6iSJCzJgfl0GFxjRVSllD1tEWqNRMgOZoOT+ QUUA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20130820; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=6PdI3E9zpyhugi7nmQXDwciegDKofKFicESl7yyogU0=; b=euhUUiPsIMvB8YaasSmgTfEK9bbsnwqN0Acpd1UZzClyVHD6Juwbo5GYWSUrS5iSxl cjCAjxlV27XW6eu6w/79iLOA+nZ0uhCpxVxKdhFXnnQIJ92wu0aR4QryijqBHFaD5dFT KUlZ4H4Zu+0sQq25IH2TispyvME5dO5W/BRdbckfOClFWtCcN9yYvPPv11ldCQyz/qfq saRaJkcKHa06JmkZcgW3VjnPGGKhXhZFk8IBrLaj+j8jlclYkyEJACXz7YXw019juL4r 9eYP22t4kFtMNyfVmymjtvzUY+BNKEFYlaaQ37jpYlOep3759yDMMCE9tZvN5oQjC45w vRdA==
X-Gm-Message-State: AKaTC024EShOYZIZIvVrXvypD23WoxL22vP8hQsndo/HIXgEiJb6ysBOlj9Ow9H/Yd7C8A==
X-Received: by with SMTP id r8mr101667566pli.23.1480721547450; Fri, 02 Dec 2016 15:32:27 -0800 (PST)
Received: from [] ([]) by with ESMTPSA id q27sm10120839pfd.49.2016. (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Dec 2016 15:32:27 -0800 (PST)
Message-ID: <>
From: Eric Dumazet <>
To: "Steinar H. Gunderson" <>
Date: Fri, 02 Dec 2016 15:31:26 -0800
In-Reply-To: <>
References: <> <> <> <> <>
Content-Type: text/plain; charset="UTF-8"
X-Mailer: Evolution 3.10.4-0ubuntu2
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Archived-At: <>
Cc: Jonathan Morton <>, Neal Cardwell <>, "" <>, bloat <>
Subject: Re: [aqm] [Bloat] TCP BBR paper is now generally available
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 02 Dec 2016 23:32:31 -0000

On Fri, 2016-12-02 at 23:40 +0100, Steinar H. Gunderson wrote:
> On Fri, Dec 02, 2016 at 05:22:23PM -0500, Neal Cardwell wrote:
> > Of course, if we find important use cases that don't work with BBR, we will
> > see what we can do to make BBR work well with them.
> I have one thing that I _wonder_ if could be BBR's fault: I run backup over
> SSH. (That would be tar + gzip + ssh.) The first full backup after I rolled
> out BBR on the server (the one sending the data) suddenly was very slow
> (~50 Mbit/sec); there was plenty of free I/O, and neither tar nor gzip
> (well, pigz) used a full core. My only remaining explanation would be that
> somehow, BBR didn't deal well with the irregular stream of data coming from
> tar. (A wget between the same machines at the same time gave 6-700 Mbit/sec.)
> I will not really blame BBR here, since I didn't take a tcpdump or have time
> to otherwise debug properly (short of eliminating the other things I already
> mentioned); most likely, it's something else. But if you've ever heard of
> others with similar issues, consider this a second report. :-)
> /* Steinar */

It would be interesting to get the chrono stats for the TCP flow, with
an updated ss/iproute2 command and the kernel patches :

efd90174167530c67a54273fd5d8369c87f9bd32 tcp: export sender limits chronographs to TCP_INFO
b0f71bd3e190df827d25d7f19bf09037567f14b7 tcp: instrument how long TCP is limited by insufficient send buffer
5615f88614a47d2b802e1d14d31b623696109276 tcp: instrument how long TCP is limited by receive window
0f87230d1a6c253681550c6064715d06a32be73d tcp: instrument how long TCP is busy sending
05b055e89121394058c75dc354e9a46e1e765579 tcp: instrument tcp sender limits chronographs