Re: [aqm] [Bloat] TCP BBR paper is now generally available

Neal Cardwell <ncardwell@google.com> Tue, 06 December 2016 21:31 UTC

Return-Path: <ncardwell@google.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F3B70129CAF for <aqm@ietfa.amsl.com>; Tue, 6 Dec 2016 13:31:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.596
X-Spam-Level:
X-Spam-Status: No, score=-5.596 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-2.896, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id idfZ8Pj6wdvz for <aqm@ietfa.amsl.com>; Tue, 6 Dec 2016 13:31:41 -0800 (PST)
Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1B0BE1294F7 for <aqm@ietf.org>; Tue, 6 Dec 2016 13:31:37 -0800 (PST)
Received: by mail-oi0-x22b.google.com with SMTP id y198so395214290oia.1 for <aqm@ietf.org>; Tue, 06 Dec 2016 13:31:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=vMer/bw2LG+zXGvhziQlM5Eis/CenBbVGcP5GqXyjQw=; b=j9dASuNegAaGsZHgzUwBhc5YVHSfWs7Ib3Pzwlos+ZZmeLDMAjfoPys0suDHRk0cvB sO6Lyj8B+WvvQR41fsXL5EnWrYY8OaxoXfWWtYiQ+OLxxu6DZw14Gqg+bCzz1249IxBg 1Ey7u6YDU/+OC231NFOuFBqliesQiwnin4KleN8GVd/T/xIgYy81fgmgDbw6xuJownDU RbdveoROk7MzgaxSM789X+IExNlBkE1LoaiqzoJCKgreSy71kmK2zUknkWS4HUVp8T5W Enp/nUZW60PzecpjA9gnVYkdBM5p1MrN9y90LnWlf5QH5nKOWq6GjnJ9OIhZ+bEvggJE yioA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=vMer/bw2LG+zXGvhziQlM5Eis/CenBbVGcP5GqXyjQw=; b=SEdzUI8wy3ouC6sN5ZowgDlDgyANRlX28Ejji4Uj49LV6ZdLplSONgMIWMy9fV32ZH oC0p3zxLB/p2F+glZrYAJZjBosECRCU7FwuNON9LcYNmQjw3xacqO2vn+FHR/wLAtGXq whZJ7N9/RPcef9D6cpPAKOp99q7vq8CZkiZLtzGppfLFLyCyc8DxPnzIWM7pA9nNqjrQ zrPj9D5x4UXvg6o2AC8GVIe8JsSMrKL2Lh1G8DxCxntsE5noqxwJGLOuvmDdEEXjN22l D8UvC8svi5pS9LEGG/uwwWaa8mFIMerDkJlG+syTmT4GeLZf6whm2lxxOaoz0TOAF/w0 CIAw==
X-Gm-Message-State: AKaTC03tO3hlaQFbPtndHje3V7NY7AJz5nYn7XwxzUL5Nq965vMJIqD1SqjKxodixFvXERjyNnzUy9QZOe7plhir
X-Received: by 10.157.6.227 with SMTP id 90mr38057719otx.73.1481059896184; Tue, 06 Dec 2016 13:31:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.202.73.195 with HTTP; Tue, 6 Dec 2016 13:31:05 -0800 (PST)
In-Reply-To: <20161206172039.GA29482@sesse.net>
References: <1480796415.18162.408.camel@edumazet-glaptop3.roam.corp.google.com> <9A821B94-143B-446F-8852-2EB0158DD57C@gmail.com> <1480799260.18162.418.camel@edumazet-glaptop3.roam.corp.google.com> <20161203213428.GB38041@sesse.net> <1480801837.18162.419.camel@edumazet-glaptop3.roam.corp.google.com> <20161203221338.GC38041@sesse.net> <1480805737.18162.425.camel@edumazet-glaptop3.roam.corp.google.com> <20161203230335.GA40411@sesse.net> <1480806940.18162.434.camel@edumazet-glaptop3.roam.corp.google.com> <1480807468.18162.436.camel@edumazet-glaptop3.roam.corp.google.com> <20161206172039.GA29482@sesse.net>
From: Neal Cardwell <ncardwell@google.com>
Date: Tue, 6 Dec 2016 16:31:05 -0500
Message-ID: <CADVnQyn=rY4_0KuaXEKgO-3E2MXgjtyNcTNQfhyKKcbJNA=wnA@mail.gmail.com>
To: "Steinar H. Gunderson" <sgunderson@bigfoot.com>
Content-Type: multipart/mixed; boundary=94eb2c04fa8201ca4a0543042107
Archived-At: <https://mailarchive.ietf.org/arch/msg/aqm/VkK-LR62uTtucl4T9m_0U7E_ukc>
Cc: Jonathan Morton <chromatix99@gmail.com>, "aqm@ietf.org" <aqm@ietf.org>, Eric Dumazet <eric.dumazet@gmail.com>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [aqm] [Bloat] TCP BBR paper is now generally available
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/aqm/>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 06 Dec 2016 21:31:43 -0000

On Tue, Dec 6, 2016 at 12:20 PM, Steinar H. Gunderson <
sgunderson@bigfoot.com> wrote:

> On Sat, Dec 03, 2016 at 03:24:28PM -0800, Eric Dumazet wrote:
> > Wait a minute. If you use fq on the receiver, then maybe your old debian
> > kernel did not backport :
> >
> > https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/
> commit/?id=9878196578286c5ed494778ada01da094377a686
>
> I upgraded to 4.7.0 (newest backport available). I can get up to ~45
> MB/sec,
> but it seems to hover more around ~22 MB/sec in this test:
>
>   http://storage.sesse.net/bbr-4.7.0.pcap


Thanks for the report, Steinar. Can you please clarify whether the BBR
behavior you are seeing is a regression vs CUBIC's behavior, or is just
mysterious?

It's hard to tell from a receiver-side trace, but this looks to me like a
send buffer limitation. The RTT looks like about 50ms, and the bandwidth is
a little over 500 Mbps, so the BDP is a little over 3 Mbytes. Looks like
most RTTs have a flight of about 2 MBytes of data, followed by a silence
suggesting perhaps the sender ran out of buffered data to send. (Screen
shot attached.)

What are your net.core.wmem_max and net.ipv4.tcp_wmem settings on the
server sending the data?

What happens if you try a bigger wmem cap, like 16 MBytes:

  sysctl -w net.core.wmem_max=16777216 net.ipv4.tcp_wmem='4096 16384
16777216'

If you happen to have access, it would be great to get a sender-side
tcpdump trace for both BBR and CUBIC.

Thanks for all your test reports!

cheers,
neal