Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?

["Zimmermann, Alexander" <[e-mail]>]

Hi Neal,

thank you so some much for this email. Especially for the summary of the
changes you folks did at Google!
See inline.

> Am 19.03.2015 um 02:20 schrieb Neal Cardwell <[e-mail]>:
> 
> On Wed, Mar 18, 2015 at 6:26 AM, Zimmermann, Alexander
> <[e-mail]> wrote:
>> Hi Michael,
>> 
>>> Am 18.03.2015 um 11:18 schrieb Scharf, Michael (Michael) <[e-mail]>:
>>> 
>>> My understanding is that TCPM recharting is still pending and IESG waits for potential feedback until March 23 [1].
>>> 
>>> The chairs are well aware of the strong support for draft-zimmermann-tcpm-cubic during IETF 91.
>>> We plan to run a formal adoption call once our charter explicitly allows adoption in TCPM.
> 
> Regarding draft-zimmermann-tcpm-cubic-00:
> 
>  https://tools.ietf.org/id/draft-zimmermann-tcpm-cubic-00.txt
> 
> I support this draft.
> 
> But I did want to note that our experience with CUBIC at Google shows
> that (1) it's important to consider stretch ACKs, and (2) it's critical to
> carefully consider the maximum rate of cwnd increase for CUBIC
> in congestion avoidance.
> 
> I would suggest the following small diffs to the draft:


Ack. If you take a look at my CUBIC presentation at the last IETF
(https://tools.ietf.org/agenda/91/slides/slides-91-tcpm-3.pdf),
one goal of this exercise will be to incorporate all changes with made
to the Linux code so far. You save me a lot of time to scan the code
for changes and compare it again w/ the draft. Thanks!

> 
> Where the draft says (in sections "3.3. Concave region" and "3.4.
> Convex region"):
> 
>  In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
>  for each received ACK.
> 
> I'd suggest the following new text in these two spots:
> 
>  In this region, cwnd MUST be incremented by
>     min((W(t+RTT) - cwnd)/cwnd, 1/2)
>  for each newly-acknowledged packet.
> 
> Detailed rationale:
> 
> (1) First is the issue of the treatment of "stretch ACKs" covering
> more than 2 packets.  The CUBIC paper and the Linux code through v3.18
> cap the de facto maximum rate of cwnd increase (in congestion
> avoidance) at 1 packet for every alternate ACK. In Google's experience
> with high-BDP paths with receiver hosts using receive
> offload/aggregation mechanisms (LRO and GRO in Linux terminology), the
> ACKs for up to 40 or more packets at a time caused this cap of "1
> packet for every alternate ACK" to result in cwnd increases that were
> too slow, leading to underutilization. So our team at Google
> contributed some changes in v3.19 that changed the rate of increase to
> be expressed in terms of packets ACKed, rather than number of ACKs.
> This approach allows full utilization even with receiver offload
> mechanisms.

We may also then recommend ABC for CUBIC when byte based stacks are
used…

> 
> (2) Second is the issue of the maximum rate of increase of cwnd. As it
> stands, this document says in section "3.4. Convex region" that:
> 
>   In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
>   for each received ACK.
> 
> There is similar language in section "3.3. Concave region".
> 
> Since the rate of increase of a cubic function can be quite large, if
> we take this language literally then in steep sections of the curve
> this can require that the sender "MUST" increase cwnd by quite
> a large amount.
> 
> I am not sure what other CUBIC implementations do, but this does not
> match the CUBIC paper or the Linux implementation. As mentioned above,
> the CUBIC paper and the Linux code through v3.18 actually cap the de
> facto maximum rate of cwnd increase (in congestion avoidance) at 1
> packet for every alternate ACK.
> 
> In our recent experiments with using CUBIC for YouTube video traffic
> using the revised, stretch-ACK-savvy CUBIC logic in Linux v3.19 we saw
> retransmit rates double due to v3.19 allowing cwnd to increase by 1
> packet for every packet ACKed.
> 
> So in Linux v4.0 we set a limit of increasing cwnd by at most 1 packet
> for every 2 packets ACKed, which restored the retransmit rates to
> their previous low levels.
> 
> For reference:
> 
> (1) Commits in Linux v3.19 to make CUBIC stretch-ACK-savvy:
> 
>  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=e73ebb0881ea5534ce606c1d71b4ac44db5c6930
>  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=814d488c61260521b1b3cc97063700a5a6667c8f
>  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9cd981dcf174d26805a032aefa791436da709bee
>  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d6b1a8a92a1417f8859a6937d2e6ffe2dfab4e6d
> 
> (2) Commits slated for Linux v4.0 to fix excessive cwnd growth in CUBIC:
> 
>  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9949afa42be0b76f5832db112ce51bb6b35b2abb
>  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d578e18ce93f5d33a7120fd57c453e22a4c0fc37
> 
> neal