Re: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

["De Schepper, Koen (Nokia - BE/Antwerp)" <[e-mail]>]

Hi Szilveszter,

First thing is to check that you can reproduce our results. With the previous (non-RTT-independent) Prague, can you confirm you get the following results?

Normally with the right coupling (p_C = p'^2 and P_L = 2*p') and if their total RTT is the same, Cubic (in Reno mode) and Prague get about the same throughput (Prague might get up to twice the throughput max). Total RTT means base RTT + all Queuing delay for each flow's round trip path. Can you confirm this, or do you see a bigger difference?

One way to check this on an equal base RTT setup is to use a singleQ PI2 which is doing the  correct marking or dropping according to the above equations, because the single Q  gives both classes the same latency (single Q = same Q delay). Another way is to use a DualQ and compensate the 15ms Classic latency with a 15ms bigger base RTT for Prague. In both cases you would see them getting around the same throughput. If the BDP (rate and/or RTT) becomes bigger, Cubic will start to get a higher throughput than Prague.

Note that if you install the latest Prague version from our L4STeam git, the default is now to be RTT independent below 25ms. So any Prague with a lower than 25 ms throughput will behave as a 25ms RTT flow (effective after 500 RTTs), and will get the same throughput as a 25ms RTT flow. You could test this with Prague on a DualPi2, when competing with a Cubic flow with a base RTT of 10ms. All RTTs (you can try even a mix of different ones in parallel) below 25ms should now get the same throughput as the Cubic one (because it has also a total RTT of 25ms).

Any other effective RTT combinations different than the equal ones should follow the "throughput ratio equals the inverse RTT ratio" rule. One exception is when non-Rtt-independent Prague flows with different RTTs ares competing on a STEP AQM. In that case the difference is bigger than that ratio, because the lowest RTTs are getting on-off marking, and behave as 1/p², instead of 1/p as they will get a more stable marking probability over their (much bigger) RTTs.

Let me know if you can reproduce the above behavior. If not, maybe try using the Linux DualPI2, and if still not as expected, then we need to dive deeper to find the reason.

Thanks,
Koen.


From: Szilveszter Nadas <[e-mail]>
Sent: Wednesday, December 2, 2020 5:05 PM
To: [e-mail]
Cc: Ferenc Fejes <[e-mail]>; Gombos Gergo <[e-mail]>; LAKI Sandor <[e-mail]>; De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]>; Tilmans, Olivier (Nokia - BE/Antwerp) <[e-mail]>
Subject: RE: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi Koen and  Olivier,

Thanks for the answers.

Some follow-up questions:

==RTT unfairness as 1:5 ratio.==
Is there a way to compensate it in the DualPI2 scheduler? My understanding was that with the right parameters it is compensated "automatically", there was good fairness among DC and Cubic flows in the original (Dual)PI2 papers. I guess that for single PI2 that was because of the shared queue. But what about DualPI2? Can it be configured in a way that  fairness remains also considering queueing for Cubic? (I vaguely remember a description of this in one of your papers, but I cannot find it now). I guess that the good fairness with DCTCP vs. Cubic was an actual anomaly, not the worse fairness with Prague vs. Cubic?

= RTT independent Prague version=
Can you provide a pointer how to configure that? Are the TCP Prague CC parameter defaults in the version we used not meaningful? Shall we change other parameters?

Cheers,
Szilveszter


-----Original Message-----
From: Tilmans, Olivier (Nokia - BE/Antwerp) <[e-mail]<mailto:[e-mail]>>
Sent: November 5, 2020 12:10
To: Szilveszter Nadas <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Ferenc Fejes <[e-mail]<mailto:[e-mail]>>; Gombos Gergo <[e-mail]<mailto:[e-mail]>>; LAKI Sandor <[e-mail]<mailto:[e-mail]>>
Subject: RE: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results



Hi Szilveszter,





 > The results are at the below link. We are somewhat surprised by much  > increased aggressiveness of TCP Prague. Can you comment on it? The settings  > and setup we used are described in the pdf (and in the original article).



Thanks for sharing these.



* Short answer:

DCTCP actively hurts itself, tripping the step AQM unnecessarily.

This is mostly benign in DC environments due to the almost non existent RTT, and small cwnd, but causes observable performance hits elsewhere.

Prague implements fixes for these issues.



* Long answer:

Please find below the main changes from dctcp to prague. The intuition behind these is to avoid triggering the step AQM part of the dual queue when there is classic traffic, or limit the extent of the received marks.

The first goal ensure that, when there is classic traffic, prague only receives marks from the coupled PI2 component, which are strong enough on their own to keep the L4S queue empty most of the time. Additionally getting marks from the step AQM means that prague is hurting itself, as it will reduce more than necessary. The second goal ensures that prague does not unnecessarily drive the AQM (either PI2 or the step) to high-marking rate; again because it will hurts its throughput and also because it will create a standing queue.



Note that depending on your experimental setups, not all the below factors will contribute.



1. Prague is always paced, regardless of the egress qdisc on the data sender,

   and paces itself at 100% of the estimated BDP.

                DCTCP is only paced when used in combination with fq, and paces at 120%

                of the BDP when that is the case.

2. Prague actively limits its gso bursts (defaults to 250us)

                DCTCP uses Linux defaults, 1ms

3. Prague implements a more accurate internal marking estimate (alpha)

                DCTCP, especially when operating with low marking probabilities, will

                tend to push down alpha to 0, delaying its response to marks (and thus

                increasing queue pressure/causing larger reduction down the line) 4. Prague carry over sub-cwnd reduction, such that multiple marks in a row

   occurring at low overall probability will eventually cause a cwnd reduction

                DCTCP's cwnd reduction code has no memory, i.e., as long as alpha is

                low enough compared to cwnd, no cwnd reduction will ever occur. This

                again drives the aqm to larger mark probabilities.



All of these points contribute to prague being more gentle towards the queue, and more reactive to the received marks (i.e., achieving a lower overall marking level)--the direct effect being smaller cwnd reduction (thus sawtooth) than DCTP, which is critical when operating at higher e2e RTTs with a shallow queue since it lowers the time to recovery.



Additionally, when operating over a long RTT path and controlling by PI2 (i.e., random marking, such that it receives marks almost every RTT--as opposed to a step where it receives nothing for several RTTs and then a RTT with most of the packets being marked), DCTCP suffers from the inaccurate/memoryless computations in 3./4. and its interactions with Linux's PRR code (which prague does not use), which can prevent it from ever reaching its expected rate.



I hope this helps to understand a bit better why prague appears to be more aggressive--it is actually the opposite: dctcp systematically hurts itself at longer base RTT, hence progressively give way as the RTT increases.

You can validate this by observing that the expected rate ratios between classic flows and prague over the dualQ match the theoretical equations (compounding the queue delay difference) even that higher base RTTs, which is not the case for DCTCP.





Best,

Olivier


From: De Schepper, Koen (Nokia - BE/Antwerp) <[e-mail]<mailto:[e-mail]>>
Sent: November 12, 2020 10:35
To: Szilveszter Nadas <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Cc: Ferenc Fejes <[e-mail]<mailto:[e-mail]>>; Gombos Gergo <[e-mail]<mailto:[e-mail]>>; LAKI Sandor <[e-mail]<mailto:[e-mail]>>
Subject: RE: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi Szilveszter,

Interesting work, thanks for sharing. There is indeed a very big difference between how DCTCP behaves today in recent kernels, than what we used in the original L4S work testing (Linux kernel 3.19). That original version was a "clean" DCTCP version that matched very well the theoretical equations: r=2/(p.RTT) for uniform stable random and r=2/(p².RTT) on/off marking. Due to many interactions, integer range constraints, pacing/GSO interactions and bugs, the performance really degraded in the recent Linux versions. These issues were fixed within the Linux Prague version, so that is why Prague should perform "better" in respect to matching the equations. It would be interesting to get an idea of the deviation from the r=2/(p.RTT) equation in your tests (relevant in DualPI2 as the coupling gives a smooth probability). Collecting the average RTT (base + queuing latency), marking probability, and rate would make it possible to evaluate this.

Main reasons why we saw deviations is when the probability varies a lot in time. As you might know DCTCP's (and Prague's) equation becomes r=2/(p².RTT) when it receives on/off marking episodes (on episodes in the order of 1 RTT, off in the order of multiple RTTs). Any not so stable marking probability results in a rate between those 2 boundaries. This is the theory, looking forward on how your practice matches this.

>From a first quick look at the results it might not deviate that much, I guess. The more aggressive part is probably due to the RTT unfairness: 1 to 5 rate ratio (with a 5ms base RTT, Classic gets a queue of 5ms+20ms target, so about 25ms and 5 times less throughput). Did you try to use the RTT independent version? If you set it to the f=max(15, RTT) mode, the minimum effective RTT becomes 15ms, so the rate ratio is 3 to 5 in that case.

Thanks and Regards,
Koen.

From: tcpPrague <[e-mail]<mailto:[e-mail]>> On Behalf Of Szilveszter Nadas
Sent: Tuesday, November 3, 2020 6:30 PM
To: [e-mail]<mailto:[e-mail]>
Cc: Ferenc Fejes <[e-mail]<mailto:[e-mail]>>; Gombos Gergo <[e-mail]<mailto:[e-mail]>>; LAKI Sandor <[e-mail]<mailto:[e-mail]>>
Subject: [tcpPrague] A Congestion Control Independent L4S Scheduler - TCP Prague preliminary results

Hi all,

During the review of our article "A Congestion Control Independent L4S Scheduler" we received comments from one of the reviewers, on why we did not also evaluate TCP Prague. So now we rerun the test cases with replacing DCTCP to TCP Prague.

The results are at the below link. We are somewhat surprised by much increased aggressiveness of TCP Prague. Can you comment on it? The settings and setup we used are described in the pdf (and in the original article).
Results: http://ppv.elte.hu/tcp-prague/
Original article (including YouTube presentation): http://ppv.elte.hu/cc-independent-l4s/

Can you comment on the results and the settings we used?

Cheers,
Szilveszter