[tcpm] Re: CWND increase in disordered state in Linux?

[Neal Cardwell <[e-mail]>]

OK, thanks for the extra details, Ingemar! Sounds like a neat project.

>From your description, it sounds like the issue is in the port of the Linux
TCP stack to the simulator. The issue should be able to be root-caused by
adding print statements to the tcp_may_raise_cwnd() code in the simulator,
printing the value of all the expressions used in that code (perhaps only
printing the state if tp->sacked_out is non-zero, to keep the output
manageable).

best regards,
neal


On Tue, Sep 3, 2024 at 2:38 PM Ingemar Johansson S <
[e-mail]> wrote:

> Hi Neal
>
>
>
> And thanks for the fast response. Now I know the expected behavior and as
> you say, it is pretty hard to get CWND and ssthresh == 7 if the CWND grows
> up to 12 MSS during the disordered state.
>
> RFC3042 is pretty clear that CWND should not be modified during disorered
> state but the text in tcp_may_raise_cwnd() made me think that it is
> perhaps allowed after all. Now I understand things better.
>
>
>
> More inline, marked [IJ]
>
>
>
> /Ingemar
>
>
>
> *From:* Neal Cardwell <[e-mail]>
> *Sent:* Tuesday, 3 September 2024 18:12
> *To:* Ingemar Johansson S <[e-mail]>; Yuchung
> Cheng <[e-mail]>
> *Cc:* [e-mail]
> *Subject:* Re: CWND increase in disordered state in Linux?
>
>
>
> On Tue, Sep 3, 2024 at 5:47 AM Ingemar Johansson S <
> [e-mail]> wrote:
>
> Hi
>
>
>
> Hi, Ingemar!
>
>
>
> Hope you are fine
>
>
>
> Yes! Hope you are fine as well. :-)
>
>
>
> I'm cc-ing +Yuchung Cheng <[e-mail]> explicitly on this thread,
> since he is the author of the tcp_may_raise_cwnd() code that Linux TCP has
> been using for about 11 years now, and he may want to add more comments or
> context.
>
>
>
> I am working on improving the TCP stack in our 5G system simulator to pass
> all relevant packetdrill tests.
>
> One thing that I I try to figure out is the behavior in CA_Disordered
> state. One such example is the prr-ss-10pkt-lost-1.pkt test.
> What I see in the current implementation is that cwnd increases from 10 to
> 12 when it is in disordered state.
> When it gets into CA_Recovery however the ssthresh should be set to 7 but
> it is set to 8 because int(12*0.7) = 8. So I guess ssthresh should be set
> based on cwnd from when the state was last in CA_Open ?.
>
>
>
> I'm surprised you are seeing cwnd increases from 10 to 12 when the
> connection is in Disorder state in that prr-ss-10pkt-lost-1.pkt test. The
> Linux TCP logic should not increase cwnd when receiving the SACKs in that
> test case (see below for the rationale). The test implicitly asserts that
> the cwnd increase upon receiving SACKs does not happen, since it explicitly
> asserts that cwnd and ssthresh are both set to 7. And AFAIK that test
> passes on upstream kernels and the kernels we use. So I'm surprised you see
> a cwnd of 8 rather than 7.
>
>
>
> Are you using this version of the test:
> https://github.com/google/packetdrill/blob/master/gtests/net/tcp/fast_recovery/prr-ss-10pkt-lost-1.pkt
> ?
>
>
>
> To help understand the discrepancy, can you please share:
>
>
>
>    1. the exact version of the packetdrill script you are using (either
>    with a git SHA1, a github URL, or attaching the script) and
>
> [IJ] The scripts are from the github repos you linked to
>
>
>
>    2. the exact Linux kernel version you are testing (with a tag name or
>    SHA1)?
>
> [IJ] 5.3, in fact it is a port of the Linux TCP stack to out Java based 5G
> system simulator, in the process I have made some errors and the packet
> drill scripts have been very helpful to identify those errors.
>
>
>
> What confuses me is that RFC3042 says that CWND should not be modified
> when in disordered state, however the explanation text in
> tcp_may_raise_cwnd() seem to indicate that CWND can increase as long as new
> data is SACKed, right?.
>
>
>
> The Linux TCP code intentionally does not exactly follow RFC3042 to the
> letter, when it comes to the limited transmit behavior. This is for good
> reason, based on decades of real-world experience.
>
> [IJ] OK, good to know
>
>
>
> The rationale for the Linux TCP tcp_may_raise_cwnd() logic is largely
> explained in Yuchung's commit messages from the 2013 code for this function:
>
>
>
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0f7cc9a3c2bd8
>
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=16edfe7ee02dd
>
>
>
>
> The full current tcp_may_raise_cwnd() logic can be browsed here:
>
>   https://elixir.bootlin.com/linux/v6.10/source/net/ipv4/tcp_input.c#L3544
>
>
>
> I will try to summarize the rationale by paraphrasing Yuchung's commit
> messages:
>
>
>
> Currently the Linux TCP stack detects reordering and avoids spurious
> retransmissions quite effectively; since roughly 2018 this is by using
> RACK-TLP: https://datatracker.ietf.org/doc/html/rfc8985
>
> However, around 2013, before RACK-TLP, and when the Linux TCP code was
> largely following the cwnd-increase dictates in RFC3042,and before
> Yuchung's commits noted above, the throughput was suboptimal for
> connections experiencing high reordering.This was because cwnd was
> increased only if the data was delivered in order. i.e., FLAG_DATA_ACKED
> was set in tcp_ack(). The more packets were reordered, the worse the
> throughput was, because with high reordering only a small fraction of ACKs
> advanced SND.UNA and allowed cwnd to increase.
>
>
> Therefore, Yuchung's commits noted above changed the logic so that:
>
>
>
> (A) when the measured degree of reordering is high (above the default
> dupthresh of 3 packets), cwnd increases whenever data is delivered
> (FLAG_FORWARD_PROGRESS is set) regardless of its ordering (regardless of
> whether packets are marked as delivered via cumulative ACKs that advance
> SND.UNA  or via SACKed out-of-order data)
>
>
>
> (B) in the common case where reordering is low, the code conservatively
> follows the cwnd increase conditions in RFC3042 and increases cwnd only on
> ordered deliveries (cumulative ACKs that advance SND.UNA).
>
>
>
> The performance difference Yuchung found in testing his changes was quite
> large: using netperf on a qdisc setup of 20Mbps bandwidth and random RTT
> from 45ms to 55ms (for reordering effect, Yuchung found his  changes
> increased TCP throughput by 20 - 25%, allowing TCP CUBIC to reach a
> throughput near the bottleneck link bandwidth.
>
> Ingemar, because the test you mention above, prr-ss-10pkt-lost-1.pkt, does
> not have reordering, the logic should be using the (B) path above, which
> matches RFC3042 behavior. So I'm surprised you are seeing the cwnd increase
> based on SACKs arriving in Disorder for that test. So I'd be curious to
> hear more details about your test and kernel to understand why your tests
> are not seeing the normal Linux TCP behavior.
>
> [IJ] Thanks for the explanation, it is very helpful. Now I just need to
> debug my code to see where it goes wrong.
>
>
>
> best regards,
>
> neal
>
>
>
>
>
>
>
> /Ingemar
>
> PS: CC:ed the tcpm group in case this topic is of more general interest
>
>