[tcpm] Re: CWND increase in disordered state in Linux?

[Ingemar Johansson S <[e-mail]>]

Hi Neal

And thanks for the fast response. Now I know the expected behavior and as you say, it is pretty hard to get CWND and ssthresh == 7 if the CWND grows up to 12 MSS during the disordered state.
RFC3042 is pretty clear that CWND should not be modified during disorered state but the text in tcp_may_raise_cwnd() made me think that it is perhaps allowed after all. Now I understand things better.

More inline, marked [IJ]

/Ingemar

From: Neal Cardwell <[e-mail]>
Sent: Tuesday, 3 September 2024 18:12
To: Ingemar Johansson S <[e-mail]>; Yuchung Cheng <[e-mail]>
Cc: [e-mail]
Subject: Re: CWND increase in disordered state in Linux?

On Tue, Sep 3, 2024 at 5:47 AM Ingemar Johansson S <[e-mail]<mailto:[e-mail]>> wrote:
Hi

Hi, Ingemar!

Hope you are fine

Yes! Hope you are fine as well. :-)

I'm cc-ing +Yuchung Cheng<mailto:[e-mail]> explicitly on this thread, since he is the author of the tcp_may_raise_cwnd() code that Linux TCP has been using for about 11 years now, and he may want to add more comments or context.

I am working on improving the TCP stack in our 5G system simulator to pass all relevant packetdrill tests.
One thing that I I try to figure out is the behavior in CA_Disordered state. One such example is the prr-ss-10pkt-lost-1.pkt test.
What I see in the current implementation is that cwnd increases from 10 to 12 when it is in disordered state.
When it gets into CA_Recovery however the ssthresh should be set to 7 but it is set to 8 because int(12*0.7) = 8. So I guess ssthresh should be set based on cwnd from when the state was last in CA_Open ?.

I'm surprised you are seeing cwnd increases from 10 to 12 when the connection is in Disorder state in that prr-ss-10pkt-lost-1.pkt test. The Linux TCP logic should not increase cwnd when receiving the SACKs in that test case (see below for the rationale). The test implicitly asserts that the cwnd increase upon receiving SACKs does not happen, since it explicitly asserts that cwnd and ssthresh are both set to 7. And AFAIK that test passes on upstream kernels and the kernels we use. So I'm surprised you see a cwnd of 8 rather than 7.

Are you using this version of the test: https://github.com/google/packetdrill/blob/master/gtests/net/tcp/fast_recovery/prr-ss-10pkt-lost-1.pkt ?

To help understand the discrepancy, can you please share:


  1.  the exact version of the packetdrill script you are using (either with a git SHA1, a github URL, or attaching the script) and
[IJ] The scripts are from the github repos you linked to


  1.  the exact Linux kernel version you are testing (with a tag name or SHA1)?
[IJ] 5.3, in fact it is a port of the Linux TCP stack to out Java based 5G system simulator, in the process I have made some errors and the packet drill scripts have been very helpful to identify those errors.

What confuses me is that RFC3042 says that CWND should not be modified when in disordered state, however the explanation text in tcp_may_raise_cwnd() seem to indicate that CWND can increase as long as new data is SACKed, right?.

The Linux TCP code intentionally does not exactly follow RFC3042 to the letter, when it comes to the limited transmit behavior. This is for good reason, based on decades of real-world experience.
[IJ] OK, good to know

The rationale for the Linux TCP tcp_may_raise_cwnd() logic is largely explained in Yuchung's commit messages from the 2013 code for this function:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0f7cc9a3c2bd8
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=16edfe7ee02dd

The full current tcp_may_raise_cwnd() logic can be browsed here:
  https://elixir.bootlin.com/linux/v6.10/source/net/ipv4/tcp_input.c#L3544

I will try to summarize the rationale by paraphrasing Yuchung's commit  messages:

Currently the Linux TCP stack detects reordering and avoids spurious retransmissions quite effectively; since roughly 2018 this is by using RACK-TLP: https://datatracker.ietf.org/doc/html/rfc8985

However, around 2013, before RACK-TLP, and when the Linux TCP code was largely following the cwnd-increase dictates in RFC3042,and before Yuchung's commits noted above, the throughput was suboptimal for connections experiencing high reordering.This was because cwnd was increased only if the data was delivered in order. i.e., FLAG_DATA_ACKED was set in tcp_ack(). The more packets were reordered, the worse the throughput was, because with high reordering only a small fraction of ACKs advanced SND.UNA and allowed cwnd to increase.

Therefore, Yuchung's commits noted above changed the logic so that:

(A) when the measured degree of reordering is high (above the default dupthresh of 3 packets), cwnd increases whenever data is delivered (FLAG_FORWARD_PROGRESS is set) regardless of its ordering (regardless of whether packets are marked as delivered via cumulative ACKs that advance SND.UNA  or via SACKed out-of-order data)

(B) in the common case where reordering is low, the code conservatively follows the cwnd increase conditions in RFC3042 and increases cwnd only on ordered deliveries (cumulative ACKs that advance SND.UNA).

The performance difference Yuchung found in testing his changes was quite large: using netperf on a qdisc setup of 20Mbps bandwidth and random RTT from 45ms to 55ms (for reordering effect, Yuchung found his  changes increased TCP throughput by 20 - 25%, allowing TCP CUBIC to reach a throughput near the bottleneck link bandwidth.
Ingemar, because the test you mention above, prr-ss-10pkt-lost-1.pkt, does not have reordering, the logic should be using the (B) path above, which matches RFC3042 behavior. So I'm surprised you are seeing the cwnd increase based on SACKs arriving in Disorder for that test. So I'd be curious to hear more details about your test and kernel to understand why your tests are not seeing the normal Linux TCP behavior.
[IJ] Thanks for the explanation, it is very helpful. Now I just need to debug my code to see where it goes wrong.

best regards,
neal



/Ingemar
PS: CC:ed the tcpm group in case this topic is of more general interest