Re: [L4s-discuss] Configuring a L4S test plant

[Sebastian Moeller <[e-mail]>]

Hi Matteo,


> On Oct 8, 2023, at 12:15, Matteo Guarna S303434 <[e-mail]> wrote:
> 
> Hi Sebastian,
> 
> Il 2023-10-07 18:59 Sebastian Moeller ha scritto:
>> Hi Matteo,
>>> On Oct 7, 2023, at 17:31, Matteo Guarna S303434 <[e-mail]> wrote:
>>> Hi Koen, thank you for being always ready to leng me your help!
>> 	[SM2] I am not Koen, and do not want to pretend I am ;) Koen has a
>> lot more in depth knowledge about the L4S internals and generally
>> considers it a good thing. In comparison I only have cursory knowledge
>> about L4S internal (and what I know does not fill me with confidence)
>> and generally am not a "fan", or as I pithily summarize it "too
>> little, too late", but that assumes it actually works as intended, and
>> I do not think your issues are caused by L4S not working, but by some
>> snag/unfortunate configuration somewhere that counters what DualQ
>> wants to do... If you have the L4S instrumentation for DualQ if you
>> look at the dash board during a test, what do you see in the
>> dash-board numbers? (I assume the dash-board is part of the repository
>> somehow...)
> 
> I am not aware of any dashboard tool for inspecting the dualQ, if anyone is aware of the existance of something along these lines and can fill me in I would be really grateful.

	[SM] I think what I am talking about is related to:
https://github.com/L4STeam/l4sdemo
From the readme:
Testbed and GUI to evaluate AQMs

This repository contains a set of tools to perform graphical evaluation of different AQMs on a testbed consisting of 5 nodes - 1 AQM node, 2 clients and 2 servers. This setup is a requirement for all the tools to work.

The testbed can be provisionned using two scripts:

	• setup_testbed.sh should be run on the AQM node
	• setup_endhosts.sh will provision the clients and servers
Both scripts assume that:

	• some environment variables are set which describe the settings. Those are listed in the environment.sh file. You can override the defaults set in environment.sh in a file named environment.local (which has to be executable).
	• All node can reach each other through a network, and all of them have a ssh server running.
The script create_network.sh can be used to create a virtualized testbed relying on libvirt/kvm.

Once the setup is complete, running run_demo.sh will start the GUI and let you measure the behavior of various AQMs and congestion controls.


Not sure whether this also works in a non virtualized setting as yours...



> 
> Matteo
> 
>>> Il 2023-10-06 17:57 Sebastian Moeller ha scritto:
>>>> Hi Matteo,
>>>>> On Oct 6, 2023, at 14:32, Matteo Guarna S303434 <[e-mail]> wrote:
>>>>> Hi Neal. Thank you for providing me with your impressions so quickly,
>>>>> On 2023-10-05 20:41 Neal Cardwell wrote:
>>>>>> Thanks for the detailed data!
>>>>>> You mention the L4S flow having a higher delay... what's the source
>>>>>> for that data?
>>>>>  [MG] I am using spindump to capture the flows passing through the router. Its code is available here: https://github.com/EricssonResearch/spindump
>>>>> I can try and produce a log of the captures, but unfortunately I have to wait until monday to access the test plant again. Still, I repeated my measurements many times over and I get a really consistent RTT measurement for Cubic each second (33.4 ms) while the Prague flows by the second vary mostly between 33.9 ms and 34.7 ms.
>>>>>> From a quick glance at the pcaps and ss data, it seems like:
>>>>>> - From the ss data, CUBIC sees RTT delays between 35ms and 53ms;
>>>>>> Prague sees RTT delays between 31ms and 35ms.
>>>>>  [MG] Your observations are much more in line with their supposed behaviour than mine. I can see that myself on the ss capture, now that you're pointing that out... Maybe Spindump is having problems with the measurements for some reason? I have to look it up I guess. Thank you!
>>>> 	[SM] Hhmm, when comparing RTTs in the two traces, Prague and Cubic
>>>> look for the longest time pretty close (Cubic has some "spikes" later
>>>> in the trace), but that should not really be if the DualQ does its
>>>> thing correctly... with DualQ as egress qdisc, how did you configure
>>>> the actual interface (how deep were the interface buffers and was BQL
>>>> active or not)?
>>>   [MG2] Honestly I do not know the depth of the buffer, I did not think about changing it so it's going the be the default size for my machine. On monday/tuesday I will be able to access it again and I will surely check, but is there a chance for it to actually affect L4S's behaviour? Like, may a buffer that's very deep result in lower than usual drops in the classic queue and in turn a smaller share for the L4S traffic? Do you have some hypoteses? Still, as soon as I am returning to the lab I will run some tests with various buffer sizes, so thenk you a lot for the suggestion.
>> 	[SM2] So this is a bit hand wavy, since I have no idea about your
>> hardware, but if you send packets to an interface it will happily
>> accepts packets as it can fit within its own internal queue and only
>> once that queue is full will it create back pressure. For something
>> like DualQ to work well it really wants/needs appropriate back
>> pressure (e.g. because the queues are kept full enough so that
>> "topping it up" will only allow a few packets to flow). If that queue
>> however gets emptied in a bursty fashion it might cause a higher than
>> expected average queueing delay which in the L-queue with its relative
>> tight reference delay (step-thresh 1ms) can cause over marking... it
>> could also cause additional delay for L-queue packets, washing out the
>> on average lower queueing delay compared to the L-queue. This is BTW a
>> good reason to measure actual end-to-end delays instead of just
>> looking at the sojourn times of the C- and L-queues, but you are
>> already doing that I believe.
>>> Regarding the BQL I honestly didn't know it could be managed, how should it be set up to allow L4S's best performance?
>> 	[SM2] Here is some old information on BQL: https://lwn.net/Articles/454390/
>>>>>> - Prague is getting about a 6% ECN mark rate, and given that it is
>>>>>> correctly converging to a rate of roughly 1/.06 - 1 ~= 15 Mbps. That
>>>>>> rate is far below its fair share of 50 Mbps. So if there is an issue
>>>>>> here, it might be in dualpi2 providing too many ECN marks to the L4S
>>>>>> flow and/or too few drops to the CUBIC flow.
>>>>>  [MG] It may well be, in fact I generate traffic with iperf3 and I can see how many retransmissions actually happen during trials of 60 seconds, where I run both flows at 100 Mbps through the bottleneck. There, while I have virtually 0 retransmissions with Prague, I can see very little retransmissions with Cubic, meaning around 20in the first second and then 1 or 2 every three seconds on average. I think this might be little too few, do you?
>>>> 	[SM] This matches what you can see in the packet captures as well if
>>>> you do a tcptrace plot, essentially zero duplicate ACKs (signs of
>>>> drops) for Prague and some for Cubic, so this is consistent...
>>>   [MG] That's reassurig to hear, but it raises a question: do you deem the number of trops in Cubic too low or in line with your expectations? The scenario consists in two 100 mbps flows on a 100 mbps bottleneck?
>>> Thank you in advance
>>> Matteo
>>>>> Thank you once again for your valuable insights
>>>>> Matteo
>>>>>> neal
>>>>>> On Thu, Oct 5, 2023 at 12:23 PM Matteo Guarna S303434
>>>>>> <[e-mail]> wrote:
>>>>>>> Hi Neal,
>>>>>>> thank you for reaching me. I executed the script on both the prague
>>>>>>> and
>>>>>>> the cubic server as you asked.
>>>>>>> The prague server has IP address 192.168.202.21, and transmits data
>>>>>>> towards 192.168.201.17
>>>>>>> The cubic server has IP address 192.168.202.22, and transmits data
>>>>>>> towards 192.168.201.18
>>>>>>> All connections lasted for 20 seconds and were established via
>>>>>>> iperf3 in
>>>>>>> reverse mode
>>>>>>> Please forgive me for having the date on the two machines out of
>>>>>>> sync
>>>>>>> (the flows had in fact started at the same time):
>>>>>>> - the transmission timestamp on the prague server begins at Thu Oct
>>>>>>> 5
>>>>>>> 2023, 05:22:50 PM CEST
>>>>>>> - the transmission timestamp on the cubic server begins at Fri Sep
>>>>>>> 29
>>>>>>> 2023, 01:37:53, CEST
>>>>>>> I am providing you with the captures as attachments to this mail: I
>>>>>>> named them with the "prague" and "cubic" suffixes after the servers
>>>>>>> where the capture took place.
>>>>>>> If you need more information please don't hesitate to contact me
>>>>>>> Best regards and thank you in advance,
>>>>>>> Matteo Guarna
>>>>>>> Il 2023-10-04 17:18 Neal Cardwell ha scritto:
>>>>>>>> Thanks for the report, Matteo.
>>>>>>>> To help debug this, could you please gather and share the
>>>>>>> following
>>>>>>>> instrumentation during one of your tests? This would need to be
>>>>>>>> collected on both data senders (servers), as root:
>>>>>>>> (while true; do date; ss -tenmoi; sleep 1; done) > /root/ss.txt &
>>>>>>>> tcpdump -w /root/dump.pcap -n -s 100 -c 1000000 host $REMOTE_HOST
>>>>>>> -i
>>>>>>>> $INTERFACE &
>>>>>>>> nstat -n; (while true; do date; nstat; sleep 1; done)  >
>>>>>>>> /root/nstat.txt &
>>>>>>>> The data should probably only be needed for the time interval
>>>>>>> starting
>>>>>>>> from before the test and ending when the flows reach steady state,
>>>>>>>> which may be 10-20 secs into the test.
>>>>>>>> thanks,
>>>>>>>> neal
>>>>>>>> On Wed, Oct 4, 2023 at 6:03 AM Sebastian Moeller
>>>>>>> <[e-mail]>
>>>>>>>> wrote:
>>>>>>>>> Hi Matteo,
>>>>>>>>>> On Oct 4, 2023, at 11:48, Matteo Guarna S303434
>>>>>>>>> <[e-mail]> wrote:
>>>>>>>>>> Hi Sebastian and thank you for your answer
>>>>>>>>>> Il 2023-10-03 16:39 Sebastian Moeller ha scritto:
>>>>>>>>>>> Hi Matteo.
>>>>>>>>>>>> On Oct 3, 2023, at 15:42, Matteo Guarna S303434
>>>>>>>>> <[e-mail]> wrote:
>>>>>>>>>>>> Greetings everyone,
>>>>>>>>>>>> I hope the question isn't too off-topic, please forgive me in
>>>>>>>>> advance if it is so.
>>>>>>>>>>>> I am still trying to perform some fairness measurements with
>>>>>>>>> both L4S and classic flow, although now on a physical test plant
>>>>>>>>> instead of a virtualized one. I'm relying on the L4STeam Github
>>>>>>>>> project for the deployment of the L4S architecture and I am
>>>>>>> looking
>>>>>>>>> for someone who's familiar with the project and might be willing
>>>>>>> to
>>>>>>>>> help me: in fact I seem not to be able to achieve the correct
>>>>>>>>> configuration.
>>>>>>>>>>>> My setup is very simple: I have four servers (two senders and
>>>>>>>>> two receivers) exchanging two traffic flows through one server
>>>>>>>>> acting as a router. One client-server pair uses Prague as CC,
>>>>>>> while
>>>>>>>>> the other uses Cubic. All servers have the patched kernel
>>>>>>> provided
>>>>>>>>> in the https://github.com/L4STeam/linux/ repository branch.
>>>>>>>>>>>> If I trigger a congestion on the router by generating both the
>>>>>>>>> Prague and the Cubic flows (let's say the flows measure 100
>>>>>>> Mbit/s
>>>>>>>>> each, and they come though a L2 switch both on the same router's
>>>>>>>>> input interface on a 1Gb Ethernet link; only a 100M link though
>>>>>>> is
>>>>>>>>> in place on the output interface towards the receivers) I see the
>>>>>>>>> L4S flow having higher delay, higher jitter and a smaller (and
>>>>>>> more
>>>>>>>>> variable) bandwidth share. The Prague share is 1/4 of the Cubic
>>>>>>>>> share. I am sending an attachment with a graphical representation
>>>>>>> of
>>>>>>>>> the scenario here described.
>>>>>>>>>>>> I configured my L4S endpoints as follows:
>>>>>>>>>>>> - I set the CC as tcp Prague (sysctl -w
>>>>>>>>> net.ipv4.tcp_congestion_control=prague)
>>>>>>>>>>>> - I set the AccEcn, even if it's not necessary apparently
>>>>>>>>> (sysctl -w net.ipv4.tcp_ecn=3)
>>>>>>>>>>>> - I disabled the required offloading capabilities on the
>>>>>>>>> endpoints (sudo ethtool -K $NETIF tso off gso off gro off lro
>>>>>>> off)
>>>>>>>>>>> [SM] I think you need to do the same on the router... or
>>>>>>>>> with your
>>>>>>>>>>> topology with running prague and cubic over separate end-points
>>>>>>>>>>> especially on the router itself. Side-node, sch_cake grew a
>>>>>>>>> split-gso
>>>>>>>>>>> mode to automatically handle this issue because it can be a bit
>>>>>>>>> of a
>>>>>>>>>>> whack-a-mole problem to make these configs stick (and in the
>>>>>>> case
>>>>>>>>> of
>>>>>>>>>>> cake the idea was to make deployment easy even for
>>>>>>> non-experts).
>>>>>>>>>> [MG] I tried as you suggested and unfortunately the situation
>>>>>>>>> remains unvaried.
>>>>>>>>> [SM2] Hmmm, that would indicate that it might not be
>>>>>>>>> "lumpyness" of inputs into the router. I guess I would take
>>>>>>> packet
>>>>>>>>> captures on both interfaces of the router to see whether there is
>>>>>>>>> any unexpected distribution of packets between both input and
>>>>>>>>> output? Also worth looking is the CPU usage on the router... we
>>>>>>>>> occasionally run into issues with aggressive?
>>>>>>>>> power/voltage/frequency scaling where a CPU might take much
>>>>>>> longer
>>>>>>>>> to wake up than expected, the L-queue with its rather low (IMHO
>>>>>>> too
>>>>>>>>> low) reference delay of 1ms would be especially sensitive to such
>>>>>>>>> issues.
>>>>>>>>> Also does your 100Mbps interface support BQL?
>>>>>>>>>> Still, I think I missed the point regarding sch_cake, could you
>>>>>>>>> explain again what it is and if and how could it be useful?
>>>>>>>>> [SM2] I am talking about Linux's cake qdisc and just as
>>>>>>>>> example, cake does not support special treatment of ECT(1) but
>>>>>>>>> implements rfc3168 ECN signaling for both ECT(0) and ECT(1). So
>>>>>>> for
>>>>>>>>> your experiments it might not be that useful (but for the fun of
>>>>>>> it,
>>>>>>>>> maybe try it as alternative for DualQ) I just mentioned it as an
>>>>>>>>> example for a qdisc that opted for not simply disabling all
>>>>>>>>> offloads. After all these offloads are quite useful, as they can
>>>>>>>>> considerably reduce the CPU of networking. (GSO/GRO work by
>>>>>>>>> ameliorating the somewhat fixed per-packet cost of Linux
>>>>>>>>> network-stack over multiple ethernet frames, as long as the
>>>>>>>>> increased deelay inherent in such bathing approaches this can
>>>>>>> help a
>>>>>>>>> lot).
>>>>>>>>>> Apologize, I guess I perfectly fit into the definition of "non
>>>>>>>>> experts". I tried to look it up on the internet but I struggled
>>>>>>> to
>>>>>>>>> find any clarification.
>>>>>>>>> [SM2] Sorry, my bad, I should have been clearer that I was
>>>>>>>>> talkning about a qdisc here, see "man tc-cake" on a sufficietly
>>>>>>>>> modern Linux system, the source code file is called sch_cake.c
>>>>>>> (see
>>>>>>>>> e.g.
>>>>>>> https://elixir.bootlin.com/linux/latest/source/net/sched/sch_cake.c)
>>>>>>>>>>>> - I configured the fair queue on the endpoints (sudo tc qdisc
>>>>>>>>> replace dev $NETIF root fq)
>>>>>>>>>>>> I configured my router as follows:
>>>>>>>>>>>> - I enabled forwarding through these interfaces to obtain the
>>>>>>>>> routing capabilities (sudo sysctl -w net.ipv4.ip_forward=1)
>>>>>>>>>>>> - I set the dualpi2 on both interfaces (sudo tc qdisc replace
>>>>>>>>> dev $NETIF root dualpi2)
>>>>>>>>>>>> I then applied the fair queue and disabled the offloading
>>>>>>>>> capabilities on both my classic endpoints to ensure that the
>>>>>>> classic
>>>>>>>>> and l4s flows act as fairly as possible, but to no avail (even
>>>>>>>>> without these precautions the results remain roughly the same).
>>>>>>>>>>> [SM] Again, I think with your topology offloads at the
>>>>>>>>> endpoints
>>>>>>>>>>> should not have much influence, but at the router the well
>>>>>>> might.
>>>>>>>>> If
>>>>>>>>>>> that turns out to help this might be explained by Prague's
>>>>>>>>> (and/or
>>>>>>>>>>> DualQ's L-queue) considerably higher sensitivity to bursty
>>>>>>>>> traffic
>>>>>>>>>>> compared to classic traffic and queue.
>>>>>>>>>>>> I am sure I am missing some important details in the setup,
>>>>>>> and
>>>>>>>>> I would really appreciate some help.
>>>>>>>>>>> [SM] To me this looks rather straight forward, and I
>>>>>>>>> probably would
>>>>>>>>>>> try something similar, but I did not actually try in practice.
>>>>>>>>>>> Regards & good luck
>>>>>>>>>>> Sebastian
>>>>>>>>>> [MG] Thanks in advance for your help, and if you have other
>>>>>>>>> tips or if you (or anyone else for that matter) are by any chance
>>>>>>>>> aware of a paper or project using the prague branch of the
>>>>>>> L4STeam
>>>>>>>>> repository, that might indeed be really helpful too.
>>>>>>>>> [SM] I am not the best/most objective person to quizz here,
>>>>>>>>> as I consider L4S in general too little too late and neither TCP
>>>>>>>>> Prague nor the DualQ AQM worth deploying in their current state
>>>>>>> (but
>>>>>>>>> that is why I consider your effort researching these admirable,
>>>>>>> both
>>>>>>>>> IMHO really need more research direly).
>>>>>>>>> I would always try to run the same tests over a bottleneck using
>>>>>>> a
>>>>>>>>> fq-scheduler, be it the all in one cake or fq_codel. Fq_codel
>>>>>>>>> actually con be configured to treat ECT(1) mire in line with what
>>>>>>>>> TCP Prague desires, so that might well be a decent starting point
>>>>>>>>> for alternative measurements....
>>>>>>>>> Regards
>>>>>>>>> Sebastian
>>>>>>>>>> My best regards to you and the community,
>>>>>>>>>> Matteo
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Matteo
>>>>>>>>>>>> P.s.
>>>>>>>>>>>> I just want to point out that by looking at the packet traces
>>>>>>>>> everything seems fine: Prague carries the ECN=1, the dualpi2
>>>>>>> marks
>>>>>>>>> packets with ECN=3, the AccEcn control signals on the ACE fields
>>>>>>> are
>>>>>>>>> coherent, and no losses occur in the Prague flow, while they do
>>>>>>>>> happen with the Cubic flow. It looks like Prague is
>>>>>>> underperforming
>>>>>>>>> for whatever reason. Furthermore, if I switch back to two Cubic
>>>>>>>>> flows I measure perfect share, equal delay and equal jitter, so
>>>>>>> it
>>>>>>>>> looks to me like there are no physical impairments on the
>>>>>>>>> testbed.<testplant_issue.pdf>--
>>>>>>>>>>>> L4s-discuss mailing list
>>>>>>>>>>>> [e-mail]
>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>>>>>> --
>>>>>>>>>> L4s-discuss mailing list
>>>>>>>>>> [e-mail]
>>>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>>>>> --
>>>>>>>>> L4s-discuss mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss--
>>>>>>> L4s-discuss mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>> --
>>>>> L4s-discuss mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>> --
>>> L4s-discuss mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
> 
> -- 
> L4s-discuss mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/l4s-discuss