Re: [L4s-discuss] Configuring a L4S test plant

[Sebastian Moeller <[e-mail]>]

Hi Matteo,


> On Oct 7, 2023, at 17:31, Matteo Guarna S303434 <[e-mail]> wrote:
> 
> Hi Koen, thank you for being always ready to leng me your help!

	[SM2] I am not Koen, and do not want to pretend I am ;) Koen has a lot more in depth knowledge about the L4S internals and generally considers it a good thing. In comparison I only have cursory knowledge about L4S internal (and what I know does not fill me with confidence) and generally am not a "fan", or as I pithily summarize it "too little, too late", but that assumes it actually works as intended, and I do not think your issues are caused by L4S not working, but by some snag/unfortunate configuration somewhere that counters what DualQ wants to do... If you have the L4S instrumentation for DualQ if you look at the dash board during a test, what do you see in the dash-board numbers? (I assume the dash-board is part of the repository somehow...)



> 
> Il 2023-10-06 17:57 Sebastian Moeller ha scritto:
>> Hi Matteo,
>>> On Oct 6, 2023, at 14:32, Matteo Guarna S303434 <[e-mail]> wrote:
>>> Hi Neal. Thank you for providing me with your impressions so quickly,
>>> On 2023-10-05 20:41 Neal Cardwell wrote:
>>>> Thanks for the detailed data!
>>>> You mention the L4S flow having a higher delay... what's the source
>>>> for that data?
>>>   [MG] I am using spindump to capture the flows passing through the router. Its code is available here: https://github.com/EricssonResearch/spindump
>>> I can try and produce a log of the captures, but unfortunately I have to wait until monday to access the test plant again. Still, I repeated my measurements many times over and I get a really consistent RTT measurement for Cubic each second (33.4 ms) while the Prague flows by the second vary mostly between 33.9 ms and 34.7 ms.
>>>> From a quick glance at the pcaps and ss data, it seems like:
>>>> - From the ss data, CUBIC sees RTT delays between 35ms and 53ms;
>>>> Prague sees RTT delays between 31ms and 35ms.
>>>   [MG] Your observations are much more in line with their supposed behaviour than mine. I can see that myself on the ss capture, now that you're pointing that out... Maybe Spindump is having problems with the measurements for some reason? I have to look it up I guess. Thank you!
>> 	[SM] Hhmm, when comparing RTTs in the two traces, Prague and Cubic
>> look for the longest time pretty close (Cubic has some "spikes" later
>> in the trace), but that should not really be if the DualQ does its
>> thing correctly... with DualQ as egress qdisc, how did you configure
>> the actual interface (how deep were the interface buffers and was BQL
>> active or not)?
>    [MG2] Honestly I do not know the depth of the buffer, I did not think about changing it so it's going the be the default size for my machine. On monday/tuesday I will be able to access it again and I will surely check, but is there a chance for it to actually affect L4S's behaviour? Like, may a buffer that's very deep result in lower than usual drops in the classic queue and in turn a smaller share for the L4S traffic? Do you have some hypoteses? Still, as soon as I am returning to the lab I will run some tests with various buffer sizes, so thenk you a lot for the suggestion.

	[SM2] So this is a bit hand wavy, since I have no idea about your hardware, but if you send packets to an interface it will happily accepts packets as it can fit within its own internal queue and only once that queue is full will it create back pressure. For something like DualQ to work well it really wants/needs appropriate back pressure (e.g. because the queues are kept full enough so that "topping it up" will only allow a few packets to flow). If that queue however gets emptied in a bursty fashion it might cause a higher than expected average queueing delay which in the L-queue with its relative tight reference delay (step-thresh 1ms) can cause over marking... it could also cause additional delay for L-queue packets, washing out the on average lower queueing delay compared to the L-queue. This is BTW a good reason to measure actual end-to-end delays instead of just looking at the sojourn times of the C- and L-queues, but you are already doing that I believe.


> 
> Regarding the BQL I honestly didn't know it could be managed, how should it be set up to allow L4S's best performance?

	[SM2] Here is some old information on BQL: https://lwn.net/Articles/454390/


>>>> - Prague is getting about a 6% ECN mark rate, and given that it is
>>>> correctly converging to a rate of roughly 1/.06 - 1 ~= 15 Mbps. That
>>>> rate is far below its fair share of 50 Mbps. So if there is an issue
>>>> here, it might be in dualpi2 providing too many ECN marks to the L4S
>>>> flow and/or too few drops to the CUBIC flow.
>>>   [MG] It may well be, in fact I generate traffic with iperf3 and I can see how many retransmissions actually happen during trials of 60 seconds, where I run both flows at 100 Mbps through the bottleneck. There, while I have virtually 0 retransmissions with Prague, I can see very little retransmissions with Cubic, meaning around 20in the first second and then 1 or 2 every three seconds on average. I think this might be little too few, do you?
>> 	[SM] This matches what you can see in the packet captures as well if
>> you do a tcptrace plot, essentially zero duplicate ACKs (signs of
>> drops) for Prague and some for Cubic, so this is consistent...
> 
>    [MG] That's reassurig to hear, but it raises a question: do you deem the number of trops in Cubic too low or in line with your expectations? The scenario consists in two 100 mbps flows on a 100 mbps bottleneck?
> 
> Thank you in advance
> 
> Matteo
> 
>>> Thank you once again for your valuable insights
>>> Matteo
>>>> neal
>>>> On Thu, Oct 5, 2023 at 12:23 PM Matteo Guarna S303434
>>>> <[e-mail]> wrote:
>>>>> Hi Neal,
>>>>> thank you for reaching me. I executed the script on both the prague
>>>>> and
>>>>> the cubic server as you asked.
>>>>> The prague server has IP address 192.168.202.21, and transmits data
>>>>> towards 192.168.201.17
>>>>> The cubic server has IP address 192.168.202.22, and transmits data
>>>>> towards 192.168.201.18
>>>>> All connections lasted for 20 seconds and were established via
>>>>> iperf3 in
>>>>> reverse mode
>>>>> Please forgive me for having the date on the two machines out of
>>>>> sync
>>>>> (the flows had in fact started at the same time):
>>>>> - the transmission timestamp on the prague server begins at Thu Oct
>>>>> 5
>>>>> 2023, 05:22:50 PM CEST
>>>>> - the transmission timestamp on the cubic server begins at Fri Sep
>>>>> 29
>>>>> 2023, 01:37:53, CEST
>>>>> I am providing you with the captures as attachments to this mail: I
>>>>> named them with the "prague" and "cubic" suffixes after the servers
>>>>> where the capture took place.
>>>>> If you need more information please don't hesitate to contact me
>>>>> Best regards and thank you in advance,
>>>>> Matteo Guarna
>>>>> Il 2023-10-04 17:18 Neal Cardwell ha scritto:
>>>>>> Thanks for the report, Matteo.
>>>>>> To help debug this, could you please gather and share the
>>>>> following
>>>>>> instrumentation during one of your tests? This would need to be
>>>>>> collected on both data senders (servers), as root:
>>>>>> (while true; do date; ss -tenmoi; sleep 1; done) > /root/ss.txt &
>>>>>> tcpdump -w /root/dump.pcap -n -s 100 -c 1000000 host $REMOTE_HOST
>>>>> -i
>>>>>> $INTERFACE &
>>>>>> nstat -n; (while true; do date; nstat; sleep 1; done)  >
>>>>>> /root/nstat.txt &
>>>>>> The data should probably only be needed for the time interval
>>>>> starting
>>>>>> from before the test and ending when the flows reach steady state,
>>>>>> which may be 10-20 secs into the test.
>>>>>> thanks,
>>>>>> neal
>>>>>> On Wed, Oct 4, 2023 at 6:03 AM Sebastian Moeller
>>>>> <[e-mail]>
>>>>>> wrote:
>>>>>>> Hi Matteo,
>>>>>>>> On Oct 4, 2023, at 11:48, Matteo Guarna S303434
>>>>>>> <[e-mail]> wrote:
>>>>>>>> Hi Sebastian and thank you for your answer
>>>>>>>> Il 2023-10-03 16:39 Sebastian Moeller ha scritto:
>>>>>>>>> Hi Matteo.
>>>>>>>>>> On Oct 3, 2023, at 15:42, Matteo Guarna S303434
>>>>>>> <[e-mail]> wrote:
>>>>>>>>>> Greetings everyone,
>>>>>>>>>> I hope the question isn't too off-topic, please forgive me in
>>>>>>> advance if it is so.
>>>>>>>>>> I am still trying to perform some fairness measurements with
>>>>>>> both L4S and classic flow, although now on a physical test plant
>>>>>>> instead of a virtualized one. I'm relying on the L4STeam Github
>>>>>>> project for the deployment of the L4S architecture and I am
>>>>> looking
>>>>>>> for someone who's familiar with the project and might be willing
>>>>> to
>>>>>>> help me: in fact I seem not to be able to achieve the correct
>>>>>>> configuration.
>>>>>>>>>> My setup is very simple: I have four servers (two senders and
>>>>>>> two receivers) exchanging two traffic flows through one server
>>>>>>> acting as a router. One client-server pair uses Prague as CC,
>>>>> while
>>>>>>> the other uses Cubic. All servers have the patched kernel
>>>>> provided
>>>>>>> in the https://github.com/L4STeam/linux/ repository branch.
>>>>>>>>>> If I trigger a congestion on the router by generating both the
>>>>>>> Prague and the Cubic flows (let's say the flows measure 100
>>>>> Mbit/s
>>>>>>> each, and they come though a L2 switch both on the same router's
>>>>>>> input interface on a 1Gb Ethernet link; only a 100M link though
>>>>> is
>>>>>>> in place on the output interface towards the receivers) I see the
>>>>>>> L4S flow having higher delay, higher jitter and a smaller (and
>>>>> more
>>>>>>> variable) bandwidth share. The Prague share is 1/4 of the Cubic
>>>>>>> share. I am sending an attachment with a graphical representation
>>>>> of
>>>>>>> the scenario here described.
>>>>>>>>>> I configured my L4S endpoints as follows:
>>>>>>>>>> - I set the CC as tcp Prague (sysctl -w
>>>>>>> net.ipv4.tcp_congestion_control=prague)
>>>>>>>>>> - I set the AccEcn, even if it's not necessary apparently
>>>>>>> (sysctl -w net.ipv4.tcp_ecn=3)
>>>>>>>>>> - I disabled the required offloading capabilities on the
>>>>>>> endpoints (sudo ethtool -K $NETIF tso off gso off gro off lro
>>>>> off)
>>>>>>>>> [SM] I think you need to do the same on the router... or
>>>>>>> with your
>>>>>>>>> topology with running prague and cubic over separate end-points
>>>>>>>>> especially on the router itself. Side-node, sch_cake grew a
>>>>>>> split-gso
>>>>>>>>> mode to automatically handle this issue because it can be a bit
>>>>>>> of a
>>>>>>>>> whack-a-mole problem to make these configs stick (and in the
>>>>> case
>>>>>>> of
>>>>>>>>> cake the idea was to make deployment easy even for
>>>>> non-experts).
>>>>>>>> [MG] I tried as you suggested and unfortunately the situation
>>>>>>> remains unvaried.
>>>>>>> [SM2] Hmmm, that would indicate that it might not be
>>>>>>> "lumpyness" of inputs into the router. I guess I would take
>>>>> packet
>>>>>>> captures on both interfaces of the router to see whether there is
>>>>>>> any unexpected distribution of packets between both input and
>>>>>>> output? Also worth looking is the CPU usage on the router... we
>>>>>>> occasionally run into issues with aggressive?
>>>>>>> power/voltage/frequency scaling where a CPU might take much
>>>>> longer
>>>>>>> to wake up than expected, the L-queue with its rather low (IMHO
>>>>> too
>>>>>>> low) reference delay of 1ms would be especially sensitive to such
>>>>>>> issues.
>>>>>>> Also does your 100Mbps interface support BQL?
>>>>>>>> Still, I think I missed the point regarding sch_cake, could you
>>>>>>> explain again what it is and if and how could it be useful?
>>>>>>> [SM2] I am talking about Linux's cake qdisc and just as
>>>>>>> example, cake does not support special treatment of ECT(1) but
>>>>>>> implements rfc3168 ECN signaling for both ECT(0) and ECT(1). So
>>>>> for
>>>>>>> your experiments it might not be that useful (but for the fun of
>>>>> it,
>>>>>>> maybe try it as alternative for DualQ) I just mentioned it as an
>>>>>>> example for a qdisc that opted for not simply disabling all
>>>>>>> offloads. After all these offloads are quite useful, as they can
>>>>>>> considerably reduce the CPU of networking. (GSO/GRO work by
>>>>>>> ameliorating the somewhat fixed per-packet cost of Linux
>>>>>>> network-stack over multiple ethernet frames, as long as the
>>>>>>> increased deelay inherent in such bathing approaches this can
>>>>> help a
>>>>>>> lot).
>>>>>>>> Apologize, I guess I perfectly fit into the definition of "non
>>>>>>> experts". I tried to look it up on the internet but I struggled
>>>>> to
>>>>>>> find any clarification.
>>>>>>> [SM2] Sorry, my bad, I should have been clearer that I was
>>>>>>> talkning about a qdisc here, see "man tc-cake" on a sufficietly
>>>>>>> modern Linux system, the source code file is called sch_cake.c
>>>>> (see
>>>>>>> e.g.
>>>>> https://elixir.bootlin.com/linux/latest/source/net/sched/sch_cake.c)
>>>>>>>>>> - I configured the fair queue on the endpoints (sudo tc qdisc
>>>>>>> replace dev $NETIF root fq)
>>>>>>>>>> I configured my router as follows:
>>>>>>>>>> - I enabled forwarding through these interfaces to obtain the
>>>>>>> routing capabilities (sudo sysctl -w net.ipv4.ip_forward=1)
>>>>>>>>>> - I set the dualpi2 on both interfaces (sudo tc qdisc replace
>>>>>>> dev $NETIF root dualpi2)
>>>>>>>>>> I then applied the fair queue and disabled the offloading
>>>>>>> capabilities on both my classic endpoints to ensure that the
>>>>> classic
>>>>>>> and l4s flows act as fairly as possible, but to no avail (even
>>>>>>> without these precautions the results remain roughly the same).
>>>>>>>>> [SM] Again, I think with your topology offloads at the
>>>>>>> endpoints
>>>>>>>>> should not have much influence, but at the router the well
>>>>> might.
>>>>>>> If
>>>>>>>>> that turns out to help this might be explained by Prague's
>>>>>>> (and/or
>>>>>>>>> DualQ's L-queue) considerably higher sensitivity to bursty
>>>>>>> traffic
>>>>>>>>> compared to classic traffic and queue.
>>>>>>>>>> I am sure I am missing some important details in the setup,
>>>>> and
>>>>>>> I would really appreciate some help.
>>>>>>>>> [SM] To me this looks rather straight forward, and I
>>>>>>> probably would
>>>>>>>>> try something similar, but I did not actually try in practice.
>>>>>>>>> Regards & good luck
>>>>>>>>> Sebastian
>>>>>>>> [MG] Thanks in advance for your help, and if you have other
>>>>>>> tips or if you (or anyone else for that matter) are by any chance
>>>>>>> aware of a paper or project using the prague branch of the
>>>>> L4STeam
>>>>>>> repository, that might indeed be really helpful too.
>>>>>>> [SM] I am not the best/most objective person to quizz here,
>>>>>>> as I consider L4S in general too little too late and neither TCP
>>>>>>> Prague nor the DualQ AQM worth deploying in their current state
>>>>> (but
>>>>>>> that is why I consider your effort researching these admirable,
>>>>> both
>>>>>>> IMHO really need more research direly).
>>>>>>> I would always try to run the same tests over a bottleneck using
>>>>> a
>>>>>>> fq-scheduler, be it the all in one cake or fq_codel. Fq_codel
>>>>>>> actually con be configured to treat ECT(1) mire in line with what
>>>>>>> TCP Prague desires, so that might well be a decent starting point
>>>>>>> for alternative measurements....
>>>>>>> Regards
>>>>>>> Sebastian
>>>>>>>> My best regards to you and the community,
>>>>>>>> Matteo
>>>>>>>>>> Regards,
>>>>>>>>>> Matteo
>>>>>>>>>> P.s.
>>>>>>>>>> I just want to point out that by looking at the packet traces
>>>>>>> everything seems fine: Prague carries the ECN=1, the dualpi2
>>>>> marks
>>>>>>> packets with ECN=3, the AccEcn control signals on the ACE fields
>>>>> are
>>>>>>> coherent, and no losses occur in the Prague flow, while they do
>>>>>>> happen with the Cubic flow. It looks like Prague is
>>>>> underperforming
>>>>>>> for whatever reason. Furthermore, if I switch back to two Cubic
>>>>>>> flows I measure perfect share, equal delay and equal jitter, so
>>>>> it
>>>>>>> looks to me like there are no physical impairments on the
>>>>>>> testbed.<testplant_issue.pdf>--
>>>>>>>>>> L4s-discuss mailing list
>>>>>>>>>> [e-mail]
>>>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>>>> --
>>>>>>>> L4s-discuss mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>>> --
>>>>>>> L4s-discuss mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss--
>>>>> L4s-discuss mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>> --
>>> L4s-discuss mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
> 
> -- 
> L4s-discuss mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/l4s-discuss