Re: [L4s-discuss] Configuring a L4S test plant

[Sebastian Moeller <[e-mail]>]

Hi Matteo,

On 6 October 2023 14:31:28 CEST, Matteo Guarna S303434 <[e-mail]> wrote:
>Hi Koen, Hi Sebastian, thank you for reaching out to me
>
>On 2023-10-06 09:09 Sebastian Moeller wrote:
>> Hi Koen,
>> 
>> 
>>> On Oct 5, 2023, at 21:43, Koen De Schepper (Nokia) <koen.de_schepper=[e-mail]> wrote:
>>> 
>>> Hi Matteo,
>>> 
>>> Does your topology have a base RTT configured on your bottleneck, or is it just LAN type of idle ping (like a few µs's)? The Prague version we have on the testing branch does not have the "below minimum window of 2" requirement implemented. This means that every Prague flow will have at least 2 packets in flight per RTT, with as a result that Prague flows on such low RTTs get a very big rate and become unresponsive when it should reduce its window below 2packets per Xµs. As a result the DualPI2 AQM will go in overload protection mode and will drop both Prague and Classic traffic,
>> 
>> 	[SM] Looking at the Prague trace, I do not seem to see dupACKs or
>> other obvious signs of drops (I might be doing it wrong though). In
>> the cubic trace I do see dupACKs, so there drops do happen...
>> 
>           [MG] Apologize for the confusion I might have created: in fact I have a server configured in bridge mode with the role of impairment, introducing 15 ms delay in each direction. It's placed after the bottlenck between the router and the clients. I am proving you an attachment with an updated schema of my testplant, including the impairment.
>
>> 
>> 
>>> where Prage with only 2 packets in flight gets a high chance of losing all packets in flight and goes in RTO (I believe 200ms timeouts),
>> 
>> 	[SM]In the prague trace there are no signs of 200ms timeouts...
>> 
>> 
>>> causing a very low average rate. Cubic has many packets in flight and in a buffer, and usually never gets RTO events, so can keep its more normal rate.
>>> 
>>> If you add a netem latency on the middle router of 5 to 20ms (which is typical in the real internet today) you will see that this issue is not appearing (at least not at 100Mbps anymore).
>
>        [MG] As I was saying previously I did, sorry for not having pointed that out earlier. On a more technical sidenote though: I used an external machine as impairment because I could not introduce delay directly on the routers' interfaces. In fact I use "tc qdisc" to configure the dualpi2 on both the router's NICs and I cannot add a second rule (e.g., netem) at the same time, but only replace the dualpi2 (which I should not obviously). Do you have some tips to have both the dualpi2 and the delay with netem running at the same time?

[SM] Assuming your router is powerful enough, us an ifb attached the ingress of each interface an instantiate netem onto those. Or us a veth pair to achieve the same (convert an ingress interface into a virtual egress so you can attach qdiscs). Whether that works well, I can not tell, as I never used netem.


>
>> 
>> 	[SM] Speaking of 20ms, is the RTT-de-bias mode on be default in
>> Prague, and if so what value is the reference RTT set to?
>> 
>> 
>>> 
>>> In our latest IETF interop we demonstrated the same problem on our PON (FttH) emulation setup, showing that this 'below minimum window of 2 packets' requirement is important to implement. But also in this case we assumed that the server is on an edge cloud, near to the Access network to have such low base latencies (of 300µs RTT).
>> 
>> 	[SM] QUESTION: since most deployed PONs seem to use some sort of
>> request/grant mechanism (e.g. DBA in GPON) is 0.3ms actually an
>> achievable RTT or would that not be closer to 1-2 ms?
>> 
>> 
>>> As a result we have implemented a new Prague update that can go to rates of below 2 packets in flight per RTT, and will be controllable down to 100kbps on any RTT below 25ms (even 0µs, and at that low rate gets theoretically 91% marking for L4S and 20% loss for Classic).
>>> So as a second testcase, you could try keeping that really low base RTT, and use the Prague kernel on the "ratebase" branch. Note this is an alpha version, not fully regression tested by us, but since you are facing the problem, you can use it. If you find any issues, contact me or Chia-Yu in CC.
>>> 
>        [MG] I might try, that thank you for the information. Still, I meant to operate at higher RTT (30 ms at the very least so that is the scenario that is more of interest to me).
>
>Thank you for your time,
>Matteo
>
>>> Success,
>>> Koen.
>>> 
>>> 
>>> -----Original Message-----
>>> From: Matteo Guarna S303434 <[e-mail]>
>>> Sent: Thursday, October 5, 2023 6:22 PM
>>> To: [e-mail]
>>> Subject: Re: [L4s-discuss] Configuring a L4S test plant
>>> 
>>> Hi Neal,
>>> 
>>> thank you for reaching me. I executed the script on both the prague and the cubic server as you asked.
>>> 
>>> The prague server has IP address 192.168.202.21, and transmits data towards 192.168.201.17 The cubic server has IP address 192.168.202.22, and transmits data towards 192.168.201.18
>>> 
>>> All connections lasted for 20 seconds and were established via iperf3 in reverse mode
>>> 
>>> Please forgive me for having the date on the two machines out of sync (the flows had in fact started at the same time):
>>> - the transmission timestamp on the prague server begins at Thu Oct 5 2023, 05:22:50 PM CEST
>>> - the transmission timestamp on the cubic server begins at Fri Sep 29 2023, 01:37:53, CEST
>>> 
>>> I am providing you with the captures as attachments to this mail: I named them with the "prague" and "cubic" suffixes after the servers where the capture took place.
>>> 
>>> 
>>> If you need more information please don't hesitate to contact me
>>> 
>>> Best regards and thank you in advance,
>>> 
>>> Matteo Guarna
>>> 
>>> 
>>> 
>>> Il 2023-10-04 17:18 Neal Cardwell ha scritto:
>>>> Thanks for the report, Matteo.
>>>> 
>>>> To help debug this, could you please gather and share the following
>>>> instrumentation during one of your tests? This would need to be
>>>> collected on both data senders (servers), as root:
>>>> 
>>>> (while true; do date; ss -tenmoi; sleep 1; done) > /root/ss.txt &
>>>> tcpdump -w /root/dump.pcap -n -s 100 -c 1000000 host $REMOTE_HOST -i
>>>> $INTERFACE & nstat -n; (while true; do date; nstat; sleep 1; done)  >
>>>> /root/nstat.txt &
>>>> 
>>>> The data should probably only be needed for the time interval starting
>>>> from before the test and ending when the flows reach steady state,
>>>> which may be 10-20 secs into the test.
>>>> 
>>>> thanks,
>>>> neal
>>>> 
>>>> On Wed, Oct 4, 2023 at 6:03 AM Sebastian Moeller <[e-mail]>
>>>> wrote:
>>>> 
>>>>> Hi Matteo,
>>>>> 
>>>>>> On Oct 4, 2023, at 11:48, Matteo Guarna S303434
>>>>> <[e-mail]> wrote:
>>>>>> 
>>>>>> Hi Sebastian and thank you for your answer
>>>>>> 
>>>>>> Il 2023-10-03 16:39 Sebastian Moeller ha scritto:
>>>>>>> Hi Matteo.
>>>>>>>> On Oct 3, 2023, at 15:42, Matteo Guarna S303434
>>>>> <[e-mail]> wrote:
>>>>>>>> Greetings everyone,
>>>>>>>> I hope the question isn't too off-topic, please forgive me in
>>>>> advance if it is so.
>>>>>>>> I am still trying to perform some fairness measurements with
>>>>> both L4S and classic flow, although now on a physical test plant
>>>>> instead of a virtualized one. I'm relying on the L4STeam Github
>>>>> project for the deployment of the L4S architecture and I am looking
>>>>> for someone who's familiar with the project and might be willing to
>>>>> help me: in fact I seem not to be able to achieve the correct
>>>>> configuration.
>>>>>>>> My setup is very simple: I have four servers (two senders and
>>>>> two receivers) exchanging two traffic flows through one server acting
>>>>> as a router. One client-server pair uses Prague as CC, while the
>>>>> other uses Cubic. All servers have the patched kernel provided in the
>>>>> https://github.com/L4STeam/linux/ repository branch.
>>>>>>>> If I trigger a congestion on the router by generating both the
>>>>> Prague and the Cubic flows (let's say the flows measure 100 Mbit/s
>>>>> each, and they come though a L2 switch both on the same router's
>>>>> input interface on a 1Gb Ethernet link; only a 100M link though is in
>>>>> place on the output interface towards the receivers) I see the L4S
>>>>> flow having higher delay, higher jitter and a smaller (and more
>>>>> variable) bandwidth share. The Prague share is 1/4 of the Cubic
>>>>> share. I am sending an attachment with a graphical representation of
>>>>> the scenario here described.
>>>>>>>> I configured my L4S endpoints as follows:
>>>>>>>> - I set the CC as tcp Prague (sysctl -w
>>>>> net.ipv4.tcp_congestion_control=prague)
>>>>>>>> - I set the AccEcn, even if it's not necessary apparently
>>>>> (sysctl -w net.ipv4.tcp_ecn=3)
>>>>>>>> - I disabled the required offloading capabilities on the
>>>>> endpoints (sudo ethtool -K $NETIF tso off gso off gro off lro off)
>>>>>>> [SM] I think you need to do the same on the router... or
>>>>> with your
>>>>>>> topology with running prague and cubic over separate end-points
>>>>>>> especially on the router itself. Side-node, sch_cake grew a
>>>>> split-gso
>>>>>>> mode to automatically handle this issue because it can be a bit
>>>>> of a
>>>>>>> whack-a-mole problem to make these configs stick (and in the case
>>>>> of
>>>>>>> cake the idea was to make deployment easy even for non-experts).
>>>>>> 
>>>>>> [MG] I tried as you suggested and unfortunately the situation
>>>>> remains unvaried.
>>>>> 
>>>>> [SM2] Hmmm, that would indicate that it might not be "lumpyness" of
>>>>> inputs into the router. I guess I would take packet captures on both
>>>>> interfaces of the router to see whether there is any unexpected
>>>>> distribution of packets between both input and output? Also worth
>>>>> looking is the CPU usage on the router... we occasionally run into
>>>>> issues with aggressive?
>>>>> power/voltage/frequency scaling where a CPU might take much longer to
>>>>> wake up than expected, the L-queue with its rather low (IMHO too
>>>>> low) reference delay of 1ms would be especially sensitive to such
>>>>> issues.
>>>>> Also does your 100Mbps interface support BQL?
>>>>> 
>>>>>> Still, I think I missed the point regarding sch_cake, could you
>>>>> explain again what it is and if and how could it be useful?
>>>>> 
>>>>> [SM2] I am talking about Linux's cake qdisc and just as example, cake
>>>>> does not support special treatment of ECT(1) but implements rfc3168
>>>>> ECN signaling for both ECT(0) and ECT(1). So for your experiments it
>>>>> might not be that useful (but for the fun of it, maybe try it as
>>>>> alternative for DualQ) I just mentioned it as an example for a qdisc
>>>>> that opted for not simply disabling all offloads. After all these
>>>>> offloads are quite useful, as they can considerably reduce the CPU of
>>>>> networking. (GSO/GRO work by ameliorating the somewhat fixed
>>>>> per-packet cost of Linux network-stack over multiple ethernet frames,
>>>>> as long as the increased deelay inherent in such bathing approaches
>>>>> this can help a lot).
>>>>> 
>>>>>> Apologize, I guess I perfectly fit into the definition of "non
>>>>> experts". I tried to look it up on the internet but I struggled to
>>>>> find any clarification.
>>>>> 
>>>>> [SM2] Sorry, my bad, I should have been clearer that I was talkning
>>>>> about a qdisc here, see "man tc-cake" on a sufficietly modern Linux
>>>>> system, the source code file is called sch_cake.c (see e.g.
>>>>> https://elixir.bootlin.com/linux/latest/source/net/sched/sch_cake.c)
>>>>> 
>>>>>> 
>>>>>>>> - I configured the fair queue on the endpoints (sudo tc qdisc
>>>>> replace dev $NETIF root fq)
>>>>>>>> I configured my router as follows:
>>>>>>>> - I enabled forwarding through these interfaces to obtain the
>>>>> routing capabilities (sudo sysctl -w net.ipv4.ip_forward=1)
>>>>>>>> - I set the dualpi2 on both interfaces (sudo tc qdisc replace
>>>>> dev $NETIF root dualpi2)
>>>>>>>> I then applied the fair queue and disabled the offloading
>>>>> capabilities on both my classic endpoints to ensure that the classic
>>>>> and l4s flows act as fairly as possible, but to no avail (even
>>>>> without these precautions the results remain roughly the same).
>>>>>>> [SM] Again, I think with your topology offloads at the
>>>>> endpoints
>>>>>>> should not have much influence, but at the router the well might.
>>>>> If
>>>>>>> that turns out to help this might be explained by Prague's
>>>>> (and/or
>>>>>>> DualQ's L-queue) considerably higher sensitivity to bursty
>>>>> traffic
>>>>>>> compared to classic traffic and queue.
>>>>>>>> I am sure I am missing some important details in the setup, and
>>>>> I would really appreciate some help.
>>>>>>> [SM] To me this looks rather straight forward, and I
>>>>> probably would
>>>>>>> try something similar, but I did not actually try in practice.
>>>>>>> Regards & good luck
>>>>>>> Sebastian
>>>>>> 
>>>>>> [MG] Thanks in advance for your help, and if you have other
>>>>> tips or if you (or anyone else for that matter) are by any chance
>>>>> aware of a paper or project using the prague branch of the L4STeam
>>>>> repository, that might indeed be really helpful too.
>>>>> 
>>>>> [SM] I am not the best/most objective person to quizz here, as I
>>>>> consider L4S in general too little too late and neither TCP Prague
>>>>> nor the DualQ AQM worth deploying in their current state (but that is
>>>>> why I consider your effort researching these admirable, both IMHO
>>>>> really need more research direly).
>>>>> 
>>>>> I would always try to run the same tests over a bottleneck using a
>>>>> fq-scheduler, be it the all in one cake or fq_codel. Fq_codel
>>>>> actually con be configured to treat ECT(1) mire in line with what TCP
>>>>> Prague desires, so that might well be a decent starting point for
>>>>> alternative measurements....
>>>>> 
>>>>> Regards
>>>>> Sebastian
>>>>> 
>>>>>> 
>>>>>> My best regards to you and the community, Matteo
>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Matteo
>>>>>>>> P.s.
>>>>>>>> I just want to point out that by looking at the packet traces
>>>>> everything seems fine: Prague carries the ECN=1, the dualpi2 marks
>>>>> packets with ECN=3, the AccEcn control signals on the ACE fields are
>>>>> coherent, and no losses occur in the Prague flow, while they do
>>>>> happen with the Cubic flow. It looks like Prague is underperforming
>>>>> for whatever reason. Furthermore, if I switch back to two Cubic flows
>>>>> I measure perfect share, equal delay and equal jitter, so it looks to
>>>>> me like there are no physical impairments on the
>>>>> testbed.<testplant_issue.pdf>--
>>>>>>>> L4s-discuss mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>> 
>>>>>> --
>>>>>> L4s-discuss mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>> 
>>>>> --
>>>>> L4s-discuss mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>> --
>>> L4s-discuss mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.