Re: [tcpm] Linux doesn’t implement RFC3465

Neal Cardwell <ncardwell@google.com> Tue, 05 November 2019 23:25 UTC

Return-Path: <ncardwell@google.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2109E1200C3 for <tcpm@ietfa.amsl.com>; Tue, 5 Nov 2019 15:25:21 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.499
X-Spam-Level:
X-Spam-Status: No, score=-17.499 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2wMSmmkP0Y-s for <tcpm@ietfa.amsl.com>; Tue, 5 Nov 2019 15:25:19 -0800 (PST)
Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 22CE2120072 for <tcpm@ietf.org>; Tue, 5 Nov 2019 15:25:19 -0800 (PST)
Received: by mail-ot1-x332.google.com with SMTP id l14so4883170oti.10 for <tcpm@ietf.org>; Tue, 05 Nov 2019 15:25:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wnhc0Sw+K3OocbjQVUPBlw+a3bNjWuXfVvTXtXEDqOI=; b=SREZdllR+irReU2F2fUCOcYQsa2hzMpArFclFPk6dYwfSod/tFHAUYm3h0Xtmfa92q /2xRJgdiozKXpTegpDw+m6zhqezvFRX3f7Y1Bu+uuudkYBv1YuivII80ijdEl+yNzOYn 5T30DXLpbFkuZUqZYiGDimrRG+oOUUeQK16HdrpAteRsxRq3GS40L6IrykwoQ2d4e6u/ ONbCCELsiKN4TMQKHR2rAqR6Z4JcEDjUDdfdbJWov1VImqK4oi//nEa5AHvN5s4AVoyD +U/dvhtedzLn3LX85YDxbsuqWVHYWkgYlhQkDAiwmKOmBXIrqJsiDCF+hda95YGSGJzg cmwg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wnhc0Sw+K3OocbjQVUPBlw+a3bNjWuXfVvTXtXEDqOI=; b=U872P9PXEmTsVdRBN5WYJ2fnO6ZEIrgcvobRnMqs/Qkulf08DfVwBd9ngbTDwied0x H7BnO2wGYFuyXCL2jneJvW9WxpOg8WKa+w62ofAZZUSgiyhMi55/IGsVSJHYyieWmy6O Gn4RUdfEZAbvW6k3Oga55RPGE9Mvk2RKIseC/OuqC0oA3yMlgOE+cT1grKb/hSE0ldXb eVCzvwGisC5TDG0dR8uzNyoKczkineBkaWqu2n4s/MRo2rVbjoaOtcCYkX3iQW5a8mYv OvIRHzN1Xq6V5UKe1UFD6B6pyoqUhVZeRCbGytIixXRT2HMmgOiB+x6h/Q0uQwKOj8f5 dUDg==
X-Gm-Message-State: APjAAAVrEWu6l+SXherbR86JhwYbf1m1Z/klPyFjPor5H//UdqcCJzQd rdydxvJj67YwPPvyJxLa5zAmyA5wM2NIBya8N2TSwAUuf1EsZA==
X-Google-Smtp-Source: APXvYqy2ccl4L+KG6zCG/nvhZ+NhPpW3FJq9MQkZpJVmlaD7BGNfaAJPPtmGerfBG1Ou7Z5qCszDb/Xt8D4wXsUPxXQ=
X-Received: by 2002:a9d:6214:: with SMTP id g20mr17626624otj.302.1572996317823; Tue, 05 Nov 2019 15:25:17 -0800 (PST)
MIME-Version: 1.0
References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com>
In-Reply-To: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com>
From: Neal Cardwell <ncardwell@google.com>
Date: Tue, 05 Nov 2019 18:25:01 -0500
Message-ID: <CADVnQynkBxTdapXN0rWOuWO3KXQ2qb6x=xhB35XrMU38JkX2DQ@mail.gmail.com>
To: Vidhi Goel <vidhi_goel=40apple.com@dmarc.ietf.org>
Cc: "tcpm@ietf.org" <tcpm@ietf.org>
Content-Type: multipart/alternative; boundary="000000000000c24b130596a1be9f"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/KE6ueJgjX7Jvi80hTn0VCQbWVHg>
Subject: Re: [tcpm] Linux doesn’t implement RFC3465
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Nov 2019 23:25:21 -0000

On Tue, Nov 5, 2019 at 5:08 PM Vidhi Goel <vidhi_goel=
40apple.com@dmarc.ietf.org> wrote:

> Hi All,
>
> RFC 3465 suggests to increase the congestion window by the number of
> previously unacknowledged bytes ACKed by each ACK limiting the increase to
> L (L = 2MSS or 1MSS in certain conditions). The limiting factor L helps to
> avoid the large bursts that could occur due to stretch ACKs, ACK loss or
> compressed ACKs (GRO/GSO).
>
> But, when I looked at Linux source code, it implements one aspect of ABC
> (byte counting to defend from ACK attacks) but doesn’t  “cap” the increase
> to L.
>  https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cong..c#L385
> <https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cong.c#L385>
>
> This is a bit confusing. How should we address this discrepancy?
>

The differences between the Linux code and the ABC RFC  (
https://tools.ietf.org/html/rfc3465 ) are intentional. The Linux TCP code
in this area is largely from:

9f9843a751d0 tcp: properly handle stretch acks in slow start
95224ac1801c Merge branch 'tcp_stretch_acks'
d6b1a8a92a14 tcp: fix timing issue in CUBIC slope calculation
9cd981dcf174 tcp: fix stretch ACK bugs in CUBIC
c22bdca94782 tcp: fix stretch ACK bugs in Reno
814d488c6126 tcp: fix the timid additive increase on stretch ACKs
e73ebb0881ea tcp: stretch ACK fixes prep

That code is based on experience at Google, both with internal traffic and
google.com and YouTube traffic. Experience was showing that preventing the
full use of ACKs in growing the cwnd in slow start and CUBIC's rapid growth
phase causes anemic growth in real-world situations.

The basic principle in the Linux code is that the sender should use the
ACKs to learn about the capacity of the path (both in volumetric and rate
dimensions), and should not ignore that information. This allows the sender
to quickly grow and achieve high throughput, even in the presence of
stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, etc.

Considering bursts is important, but that can be tackled as an orthogonal
issue. Bursts are avoided in the Linux TCP ecosystem by the combination of
TSO autosizing, pacing, TSQ, and fair queueing.

Here is a summary of the Linux TCP design in this area:
  https://netdevconf.info/1.2/papers/bbr-netdev-1.2.new.new.pdf

best,
neal