Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?

Neal Cardwell <> Thu, 19 March 2015 01:21 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id 85E5B1AC3CC for <>; Wed, 18 Mar 2015 18:21:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.389
X-Spam-Status: No, score=-1.389 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id abpGSUy_W9SQ for <>; Wed, 18 Mar 2015 18:21:27 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4003:c01::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 79C7E1AC407 for <>; Wed, 18 Mar 2015 18:20:55 -0700 (PDT)
Received: by obdfc2 with SMTP id fc2so44118746obd.3 for <>; Wed, 18 Mar 2015 18:20:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XwrwfNB7mwalcm5tbJ1BfR4dGhS2ZZnrRZbVnC7W5RY=; b=T9FMlyQ6Dn9LlXyNxbf2KhwV7kT69eBleLQ1w/wruNu5D6Ndqu7G7WJWkwnfx+rcVb 3jgTPr8WtiyQtPjz8g0xc2oPh7XulUnPqnGm4tPeAvhm1Pl5dC9YFj3kstnuM6qhahte /64zDWnMjigHEUoNyO5BFV64ypE/Cx8zk1mrXX1wV3/CJrwYJiPe+Tv3mIHCl6xMQn/O P2GpuYM9z78KYVgGukd44ZsjlwtaWmee3KBciMTLMI+5Zo0DVfsNcJPTJdXaNNARR7Ac oPu+Po3E7DoDqHswEjwc5zXkbD9dXtgyOVVC1qPKkr6mvP/xCmXvY3+FMbV8kB4svyHP ECMA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=XwrwfNB7mwalcm5tbJ1BfR4dGhS2ZZnrRZbVnC7W5RY=; b=QbG1w3lULztktuiVADcWxYVtAQAfhotl6WlPdUMUeK8n96cuwdAP7xS3o3amIj4y4t 4E50mcKi0IDNzQ486CIaHwXSSKDdSyjgpATOQKgHsM+2kAPjiiLHG/itJ9mxJ6RtDpSn +mrJsRYfUCfXNyJiUbgnlwXL2tUJ8bFwVpz/ig8ClFpaw0DWAZthEHudDBprbSccowTI jD515KK8T44PbmO7HMSTt4hyf/2gVFM1v2FtJ8WRKrhMXFZkWWaG5+EHlK/8f5gg5+67 6jFdwA6f1RvR2btYleyTuOdLphxv1yMmtKYPVMdaa2LCo4tYBcUrG0F0StIjvoX5DLuA QhUg==
X-Gm-Message-State: ALoCoQkzWcsZePPXwps74F0jA3A9mILDa1Bn629oWBZ4z0YgfZ+Y3xh0iYbMwR9Lqndh5WBGPRy2
MIME-Version: 1.0
X-Received: by with SMTP id e8mr8899427oig.112.1426728054880; Wed, 18 Mar 2015 18:20:54 -0700 (PDT)
Received: by with HTTP; Wed, 18 Mar 2015 18:20:54 -0700 (PDT)
In-Reply-To: <>
References: <> <> <>
Date: Wed, 18 Mar 2015 21:20:54 -0400
Message-ID: <>
From: Neal Cardwell <>
To: "Zimmermann, Alexander" <>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <>
Cc: "" <>, Eric Dumazet <>, "" <>
Subject: Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 19 Mar 2015 01:21:28 -0000

On Wed, Mar 18, 2015 at 6:26 AM, Zimmermann, Alexander
<> wrote:
> Hi Michael,
>> Am 18.03.2015 um 11:18 schrieb Scharf, Michael (Michael) <>:
>> My understanding is that TCPM recharting is still pending and IESG waits for potential feedback until March 23 [1].
>> The chairs are well aware of the strong support for draft-zimmermann-tcpm-cubic during IETF 91.
>> We plan to run a formal adoption call once our charter explicitly allows adoption in TCPM.

Regarding draft-zimmermann-tcpm-cubic-00:

I support this draft.

But I did want to note that our experience with CUBIC at Google shows
that (1) it's important to consider stretch ACKs, and (2) it's critical to
carefully consider the maximum rate of cwnd increase for CUBIC
in congestion avoidance.

I would suggest the following small diffs to the draft:

Where the draft says (in sections "3.3. Concave region" and "3.4.
Convex region"):

  In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
  for each received ACK.

I'd suggest the following new text in these two spots:

  In this region, cwnd MUST be incremented by
     min((W(t+RTT) - cwnd)/cwnd, 1/2)
  for each newly-acknowledged packet.

Detailed rationale:

(1) First is the issue of the treatment of "stretch ACKs" covering
more than 2 packets.  The CUBIC paper and the Linux code through v3.18
cap the de facto maximum rate of cwnd increase (in congestion
avoidance) at 1 packet for every alternate ACK. In Google's experience
with high-BDP paths with receiver hosts using receive
offload/aggregation mechanisms (LRO and GRO in Linux terminology), the
ACKs for up to 40 or more packets at a time caused this cap of "1
packet for every alternate ACK" to result in cwnd increases that were
too slow, leading to underutilization. So our team at Google
contributed some changes in v3.19 that changed the rate of increase to
be expressed in terms of packets ACKed, rather than number of ACKs.
This approach allows full utilization even with receiver offload

(2) Second is the issue of the maximum rate of increase of cwnd. As it
stands, this document says in section "3.4. Convex region" that:

   In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
   for each received ACK.

There is similar language in section "3.3. Concave region".

Since the rate of increase of a cubic function can be quite large, if
we take this language literally then in steep sections of the curve
this can require that the sender "MUST" increase cwnd by quite
a large amount.

I am not sure what other CUBIC implementations do, but this does not
match the CUBIC paper or the Linux implementation. As mentioned above,
the CUBIC paper and the Linux code through v3.18 actually cap the de
facto maximum rate of cwnd increase (in congestion avoidance) at 1
packet for every alternate ACK.

In our recent experiments with using CUBIC for YouTube video traffic
using the revised, stretch-ACK-savvy CUBIC logic in Linux v3.19 we saw
retransmit rates double due to v3.19 allowing cwnd to increase by 1
packet for every packet ACKed.

So in Linux v4.0 we set a limit of increasing cwnd by at most 1 packet
for every 2 packets ACKed, which restored the retransmit rates to
their previous low levels.

For reference:

(1) Commits in Linux v3.19 to make CUBIC stretch-ACK-savvy:

(2) Commits slated for Linux v4.0 to fix excessive cwnd growth in CUBIC: