Re: [tcpm] Linux doesn’t implement RFC3465

[Neal Cardwell <[e-mail]>]

I would vote that the guidance should be something like:

  On an ACK, don't increase the cwnd by more than the amount of data ACKed.

This implies:

 o slow-start is limited to doubling each round trip

 o the Savage ACK division attacks are squashed

I don't think limiting cwnd growth to some constant L less than cwnd is the
best road to go down, since:

(a) At high speeds (at least >10Gbps) TCP has needed to operate in
TSO/GRO/LRO/GRO aggregates of 64KBytes for a decade or more, and is now
moving on to larger aggregates to go beyond 100Gbps (see Eric Dumazet's
nice talk at the Linux Netdev conference a few weeks ago). Having a
constant L is not scalable or future-proof. I would imagine any constant L
lower than 64KBytes will be ignored, in practice, with good reason. A
constant L bigger than 64KBytes will be out of date soon, if it's not
already.

(b) If we're worried about bursts (which we should be), then we should
recommend pacing. And RFC 7661 already does that:
  https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2

  "A TCP sender in the non-validated phase SHOULD control the maximum
   burst size, e.g., using a rate-based pacing algorithm in which a
   sender paces out the cwnd over its estimate of the RTT, or some
   other method, to prevent many segments being transmitted
   contiguously at line-rate."

  So a sender that follows RFC 7661 already has a solution for bursts:
pacing.

(b) If an implementation ignores RFC 7661 and does not do pacing, then
limiting cwnd *growth* to L per ACK does not prevent bursts. A sender that
ignores RFC 7661 will be frequently restarting from idle to blast a full
cwnd at line rate. Those restart-from-idle cwnd bursts are already
super-common in some of the most common types of TCP traffic: web, RPC,
streaming video. Having a particular L limit for cwnd growth will not save
us from those cwnd-magnitude bursts.

Just my two cents on that,
neal


On Wed, Jul 28, 2021 at 7:20 PM Yuchung Cheng <[e-mail]> wrote:

> Thank you Vidhi and Mark for supporting an ABC update.
>
> To give this a start, my recommended update is:
>
> 1) if the sender uses some form of pacing to send data packets, L is
> RECOMMENDED to be the effective window worth of packets. Pacing here refers
> to spread packet transmission following a rate based on the congestion
> window and round trip. Additionally if SACK is supported, the same L
> applies in slow start after RTO
>
> 2) otherwise L is RECOMMENDED to be 8
>
> Thoughts?
>
>
> On Wed, Jul 28, 2021 at 2:13 PM Vidhi Goel <vidhi_goel=
> [e-mail]> wrote:
>
>> Resurrecting the 3465 thread.
>>
>> In the TCPM meeting at IETF 111, we discussed about this issue of L=2
>> which is a MUST in RFC 3465. This is a very strict requirement and stacks
>> like Linux already doesn’t follow it.
>>
>> The basic principle in the Linux code is that the sender should use the
>> ACKs to learn about the capacity of the path (both in volumetric and rate
>> dimensions), and should not ignore that information. This allows the sender
>> to quickly grow and achieve high throughput, even in the presence of
>> stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, etc.
>>
>> Considering bursts is important, but that can be tackled as an orthogonal
>> issue. Bursts are avoided in the Linux TCP ecosystem by the combination of
>> TSO autosizing, pacing, TSQ, and fair queueing.
>>
>>
>> As Neal described, the congestion controller should use the information
>> in stretch ACKs to increase its congestion window so that it correctly
>> adjusts the *cwnd* based on available link capacity.
>> Burstiness is an orthogonal issue which can be solved by pacing.
>>
>> QUIC loss recovery (RFC 9002) also follows this approach.
>>
>> *Mark*,
>> As a lot of transport and congestion control drafts reference RFC 3465,
>> do you think we should update this RFC to reflect the current deployment?
>> This would also be useful for someone who is just starting with a new
>> implementation.
>>
>> Thanks,
>> Vidhi
>>
>> On Nov 27, 2019, at 5:14 AM, Mark Allman <[e-mail]> wrote:
>>
>>
>> +Mark Allman
>>
>>
>> Just to clear it up, I *was* at BBN long ago when the ABC document
>> was written.  It's a cool place to work.  I recommend it.  But, I
>> now hang out at ICSI.
>>
>> I believe that ABC was written to solve the problem with ACK
>> counting by counting the number of bytes acknowledged for
>> misbehaving receivers. Limiting the increase to 2*MSS was a good
>> solution to avoid bursts at the time.
>>
>>
>> The main motivation behind ABC was to counteract delayed ACKs.  The
>> common approach at the time was to just bump cwnd by one MSS every
>> time an ACK rolled in.  So, if an ACK covered two segments because
>> the receiver was delaying the ACKs then the growth rate during slow
>> start was 1.5x per RTT instead of the 2x that was really
>> envisioned.  During congestion avoidance the growth was 1 MSS every
>> 2 RTTs instead of the envisioned every RTT.
>>
>> A secondary motivation was to counteract these ACK division attacks
>> that Savage taught us about.  I.e., we could ACK an MSS-sized packet
>> one byte at a time and the sender would then increase the cwnd by
>> MSS*MSS bytes in the prevalent ACK counting scheme (i.e., cwnd would
>> get bumped by MSS bytes for every ACK).
>>
>> The limit has two roots ...
>>
>> (1) The limit is important in slow starts that follow an RTO.  As
>>    the RFC discusses, in this case we might retransmit a single
>>    packet and this will cause the receiver's window to slide a
>>    great deal.  Therefore, an ACK may indicate that a ton of data
>>    has left the network, but that isn't really the case.  So, we
>>    don't want to increase the cwnd based on all the new bytes
>>    ACKed.
>>
>>    I have since mostly decided that this use of L is crude.
>>    Probably there is a more elegant way to do it by using the
>>    scoreboard and the SACK information to get a better
>>    understanding of what left the network and when.  That said, in
>>    this case L is simple and probably about right most of the
>>    time.
>>
>> (2) I think there was some general conservativeness to bursts and
>>    using L everywhere quelled some of the worry.  Here L=2 was used
>>    to exactly offset delayed ACKs.
>>
>> I agree that increasing the congestion window and controlling the
>> burst rate are orthogonal issues.
>>
>>
>> Yes.  In fact, we did subsequent research on mitigating bursts
>> because we never really thought of ABC as somehow the way to control
>> bursts (research papers available, but never went into RFCs).
>>
>> And, in a world that leverages stretch ACKs as a routine I think
>> Linux's approach of not using an L may well be correct.  Documenting
>> that and the reasoning behind it in modern networks seems useful to
>> me.
>>
>> allman
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>