[tcpm] Re: CWND increase in disordered state in Linux?

[Neal Cardwell <[e-mail]>]

On Tue, Sep 3, 2024 at 5:47 AM Ingemar Johansson S <
[e-mail]> wrote:

> Hi
>

Hi, Ingemar!


> Hope you are fine
>

Yes! Hope you are fine as well. :-)

I'm cc-ing +Yuchung Cheng <[e-mail]> explicitly on this thread,
since he is the author of the tcp_may_raise_cwnd() code that Linux TCP has
been using for about 11 years now, and he may want to add more comments or
context.

I am working on improving the TCP stack in our 5G system simulator to pass
> all relevant packetdrill tests.
>
> One thing that I I try to figure out is the behavior in CA_Disordered
> state. One such example is the prr-ss-10pkt-lost-1.pkt test.
> What I see in the current implementation is that cwnd increases from 10 to
> 12 when it is in disordered state.
> When it gets into CA_Recovery however the ssthresh should be set to 7 but
> it is set to 8 because int(12*0.7) = 8. So I guess ssthresh should be set
> based on cwnd from when the state was last in CA_Open ?.
>

I'm surprised you are seeing cwnd increases from 10 to 12 when the
connection is in Disorder state in that prr-ss-10pkt-lost-1.pkt test. The
Linux TCP logic should not increase cwnd when receiving the SACKs in that
test case (see below for the rationale). The test implicitly asserts that
the cwnd increase upon receiving SACKs does not happen, since it explicitly
asserts that cwnd and ssthresh are both set to 7. And AFAIK that test
passes on upstream kernels and the kernels we use. So I'm surprised you see
a cwnd of 8 rather than 7.

Are you using this version of the test:
https://github.com/google/packetdrill/blob/master/gtests/net/tcp/fast_recovery/prr-ss-10pkt-lost-1.pkt
?

To help understand the discrepancy, can you please share:

(a) the exact version of the packetdrill script you are using (either with
a git SHA1, a github URL, or attaching the script) and

(b) the exact Linux kernel version you are testing (with a tag name or
SHA1)?


> What confuses me is that RFC3042 says that CWND should not be modified
> when in disordered state, however the explanation text in
> tcp_may_raise_cwnd() seem to indicate that CWND can increase as long as new
> data is SACKed, right?.
>

The Linux TCP code intentionally does not exactly follow RFC3042 to the
letter, when it comes to the limited transmit behavior. This is for good
reason, based on decades of real-world experience.

The rationale for the Linux TCP tcp_may_raise_cwnd() logic is largely
explained in Yuchung's commit messages from the 2013 code for this function:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0f7cc9a3c2bd8
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=16edfe7ee02dd


The full current tcp_may_raise_cwnd() logic can be browsed here:
  https://elixir.bootlin.com/linux/v6.10/source/net/ipv4/tcp_input.c#L3544

I will try to summarize the rationale by paraphrasing Yuchung's commit
messages:

Currently the Linux TCP stack detects reordering and avoids spurious
retransmissions quite effectively; since roughly 2018 this is by using
RACK-TLP: https://datatracker.ietf.org/doc/html/rfc8985

However, around 2013, before RACK-TLP, and when the Linux TCP code was
largely following the cwnd-increase dictates in RFC3042,and before
Yuchung's commits noted above, the throughput was suboptimal for
connections experiencing high reordering.This was because cwnd was
increased only if the data was delivered in order. i.e., FLAG_DATA_ACKED
was set in tcp_ack(). The more packets were reordered, the worse the
throughput was, because with high reordering only a small fraction of ACKs
advanced SND.UNA and allowed cwnd to increase.

Therefore, Yuchung's commits noted above changed the logic so that:

(A) when the measured degree of reordering is high (above the default
dupthresh of 3 packets), cwnd increases whenever data is delivered
(FLAG_FORWARD_PROGRESS is set) regardless of its ordering (regardless of
whether packets are marked as delivered via cumulative ACKs that advance
SND.UNA  or via SACKed out-of-order data)

(B) in the common case where reordering is low, the code conservatively
follows the cwnd increase conditions in RFC3042 and increases cwnd only on
ordered deliveries (cumulative ACKs that advance SND.UNA).

The performance difference Yuchung found in testing his changes was quite
large: using netperf on a qdisc setup of 20Mbps bandwidth and random RTT
from 45ms to 55ms (for reordering effect, Yuchung found his  changes
increased TCP throughput by 20 - 25%, allowing TCP CUBIC to reach a
throughput near the bottleneck link bandwidth.

Ingemar, because the test you mention above, prr-ss-10pkt-lost-1.pkt, does
not have reordering, the logic should be using the (B) path above, which
matches RFC3042 behavior. So I'm surprised you are seeing the cwnd increase
based on SACKs arriving in Disorder for that test. So I'd be curious to
hear more details about your test and kernel to understand why your tests
are not seeing the normal Linux TCP behavior.

best regards,
neal



>
>
> /Ingemar
>
> PS: CC:ed the tcpm group in case this topic is of more general interest
>