Re: [L4s-discuss] Configuring a L4S test plant

[Matteo Guarna S303434 <[e-mail]>]

Hi Sebastian,

Il 2023-10-07 18:59 Sebastian Moeller ha scritto:
> Hi Matteo,
> 
> 
>> On Oct 7, 2023, at 17:31, Matteo Guarna S303434 
>> <[e-mail]> wrote:
>> 
>> Hi Koen, thank you for being always ready to leng me your help!
> 
> 	[SM2] I am not Koen, and do not want to pretend I am ;) Koen has a
> lot more in depth knowledge about the L4S internals and generally
> considers it a good thing. In comparison I only have cursory knowledge
> about L4S internal (and what I know does not fill me with confidence)
> and generally am not a "fan", or as I pithily summarize it "too
> little, too late", but that assumes it actually works as intended, and
> I do not think your issues are caused by L4S not working, but by some
> snag/unfortunate configuration somewhere that counters what DualQ
> wants to do... If you have the L4S instrumentation for DualQ if you
> look at the dash board during a test, what do you see in the
> dash-board numbers? (I assume the dash-board is part of the repository
> somehow...)
> 
> 

I am not aware of any dashboard tool for inspecting the dualQ, if anyone 
is aware of the existance of something along these lines and can fill me 
in I would be really grateful.

Matteo

> 
>> 
>> Il 2023-10-06 17:57 Sebastian Moeller ha scritto:
>>> Hi Matteo,
>>>> On Oct 6, 2023, at 14:32, Matteo Guarna S303434 
>>>> <[e-mail]> wrote:
>>>> Hi Neal. Thank you for providing me with your impressions so 
>>>> quickly,
>>>> On 2023-10-05 20:41 Neal Cardwell wrote:
>>>>> Thanks for the detailed data!
>>>>> You mention the L4S flow having a higher delay... what's the source
>>>>> for that data?
>>>>   [MG] I am using spindump to capture the flows passing through the 
>>>> router. Its code is available here: 
>>>> https://github.com/EricssonResearch/spindump
>>>> I can try and produce a log of the captures, but unfortunately I 
>>>> have to wait until monday to access the test plant again. Still, I 
>>>> repeated my measurements many times over and I get a really 
>>>> consistent RTT measurement for Cubic each second (33.4 ms) while the 
>>>> Prague flows by the second vary mostly between 33.9 ms and 34.7 ms.
>>>>> From a quick glance at the pcaps and ss data, it seems like:
>>>>> - From the ss data, CUBIC sees RTT delays between 35ms and 53ms;
>>>>> Prague sees RTT delays between 31ms and 35ms.
>>>>   [MG] Your observations are much more in line with their supposed 
>>>> behaviour than mine. I can see that myself on the ss capture, now 
>>>> that you're pointing that out... Maybe Spindump is having problems 
>>>> with the measurements for some reason? I have to look it up I guess. 
>>>> Thank you!
>>> 	[SM] Hhmm, when comparing RTTs in the two traces, Prague and Cubic
>>> look for the longest time pretty close (Cubic has some "spikes" later
>>> in the trace), but that should not really be if the DualQ does its
>>> thing correctly... with DualQ as egress qdisc, how did you configure
>>> the actual interface (how deep were the interface buffers and was BQL
>>> active or not)?
>>    [MG2] Honestly I do not know the depth of the buffer, I did not 
>> think about changing it so it's going the be the default size for my 
>> machine. On monday/tuesday I will be able to access it again and I 
>> will surely check, but is there a chance for it to actually affect 
>> L4S's behaviour? Like, may a buffer that's very deep result in lower 
>> than usual drops in the classic queue and in turn a smaller share for 
>> the L4S traffic? Do you have some hypoteses? Still, as soon as I am 
>> returning to the lab I will run some tests with various buffer sizes, 
>> so thenk you a lot for the suggestion.
> 
> 	[SM2] So this is a bit hand wavy, since I have no idea about your
> hardware, but if you send packets to an interface it will happily
> accepts packets as it can fit within its own internal queue and only
> once that queue is full will it create back pressure. For something
> like DualQ to work well it really wants/needs appropriate back
> pressure (e.g. because the queues are kept full enough so that
> "topping it up" will only allow a few packets to flow). If that queue
> however gets emptied in a bursty fashion it might cause a higher than
> expected average queueing delay which in the L-queue with its relative
> tight reference delay (step-thresh 1ms) can cause over marking... it
> could also cause additional delay for L-queue packets, washing out the
> on average lower queueing delay compared to the L-queue. This is BTW a
> good reason to measure actual end-to-end delays instead of just
> looking at the sojourn times of the C- and L-queues, but you are
> already doing that I believe.
> 
> 
>> 
>> Regarding the BQL I honestly didn't know it could be managed, how 
>> should it be set up to allow L4S's best performance?
> 
> 	[SM2] Here is some old information on BQL: 
> https://lwn.net/Articles/454390/
> 
> 
>>>>> - Prague is getting about a 6% ECN mark rate, and given that it is
>>>>> correctly converging to a rate of roughly 1/.06 - 1 ~= 15 Mbps. 
>>>>> That
>>>>> rate is far below its fair share of 50 Mbps. So if there is an 
>>>>> issue
>>>>> here, it might be in dualpi2 providing too many ECN marks to the 
>>>>> L4S
>>>>> flow and/or too few drops to the CUBIC flow.
>>>>   [MG] It may well be, in fact I generate traffic with iperf3 and I 
>>>> can see how many retransmissions actually happen during trials of 60 
>>>> seconds, where I run both flows at 100 Mbps through the bottleneck. 
>>>> There, while I have virtually 0 retransmissions with Prague, I can 
>>>> see very little retransmissions with Cubic, meaning around 20in the 
>>>> first second and then 1 or 2 every three seconds on average. I think 
>>>> this might be little too few, do you?
>>> 	[SM] This matches what you can see in the packet captures as well if
>>> you do a tcptrace plot, essentially zero duplicate ACKs (signs of
>>> drops) for Prague and some for Cubic, so this is consistent...
>> 
>>    [MG] That's reassurig to hear, but it raises a question: do you 
>> deem the number of trops in Cubic too low or in line with your 
>> expectations? The scenario consists in two 100 mbps flows on a 100 
>> mbps bottleneck?
>> 
>> Thank you in advance
>> 
>> Matteo
>> 
>>>> Thank you once again for your valuable insights
>>>> Matteo
>>>>> neal
>>>>> On Thu, Oct 5, 2023 at 12:23 PM Matteo Guarna S303434
>>>>> <[e-mail]> wrote:
>>>>>> Hi Neal,
>>>>>> thank you for reaching me. I executed the script on both the 
>>>>>> prague
>>>>>> and
>>>>>> the cubic server as you asked.
>>>>>> The prague server has IP address 192.168.202.21, and transmits 
>>>>>> data
>>>>>> towards 192.168.201.17
>>>>>> The cubic server has IP address 192.168.202.22, and transmits data
>>>>>> towards 192.168.201.18
>>>>>> All connections lasted for 20 seconds and were established via
>>>>>> iperf3 in
>>>>>> reverse mode
>>>>>> Please forgive me for having the date on the two machines out of
>>>>>> sync
>>>>>> (the flows had in fact started at the same time):
>>>>>> - the transmission timestamp on the prague server begins at Thu 
>>>>>> Oct
>>>>>> 5
>>>>>> 2023, 05:22:50 PM CEST
>>>>>> - the transmission timestamp on the cubic server begins at Fri Sep
>>>>>> 29
>>>>>> 2023, 01:37:53, CEST
>>>>>> I am providing you with the captures as attachments to this mail: 
>>>>>> I
>>>>>> named them with the "prague" and "cubic" suffixes after the 
>>>>>> servers
>>>>>> where the capture took place.
>>>>>> If you need more information please don't hesitate to contact me
>>>>>> Best regards and thank you in advance,
>>>>>> Matteo Guarna
>>>>>> Il 2023-10-04 17:18 Neal Cardwell ha scritto:
>>>>>>> Thanks for the report, Matteo.
>>>>>>> To help debug this, could you please gather and share the
>>>>>> following
>>>>>>> instrumentation during one of your tests? This would need to be
>>>>>>> collected on both data senders (servers), as root:
>>>>>>> (while true; do date; ss -tenmoi; sleep 1; done) > /root/ss.txt &
>>>>>>> tcpdump -w /root/dump.pcap -n -s 100 -c 1000000 host $REMOTE_HOST
>>>>>> -i
>>>>>>> $INTERFACE &
>>>>>>> nstat -n; (while true; do date; nstat; sleep 1; done)  >
>>>>>>> /root/nstat.txt &
>>>>>>> The data should probably only be needed for the time interval
>>>>>> starting
>>>>>>> from before the test and ending when the flows reach steady 
>>>>>>> state,
>>>>>>> which may be 10-20 secs into the test.
>>>>>>> thanks,
>>>>>>> neal
>>>>>>> On Wed, Oct 4, 2023 at 6:03 AM Sebastian Moeller
>>>>>> <[e-mail]>
>>>>>>> wrote:
>>>>>>>> Hi Matteo,
>>>>>>>>> On Oct 4, 2023, at 11:48, Matteo Guarna S303434
>>>>>>>> <[e-mail]> wrote:
>>>>>>>>> Hi Sebastian and thank you for your answer
>>>>>>>>> Il 2023-10-03 16:39 Sebastian Moeller ha scritto:
>>>>>>>>>> Hi Matteo.
>>>>>>>>>>> On Oct 3, 2023, at 15:42, Matteo Guarna S303434
>>>>>>>> <[e-mail]> wrote:
>>>>>>>>>>> Greetings everyone,
>>>>>>>>>>> I hope the question isn't too off-topic, please forgive me in
>>>>>>>> advance if it is so.
>>>>>>>>>>> I am still trying to perform some fairness measurements with
>>>>>>>> both L4S and classic flow, although now on a physical test plant
>>>>>>>> instead of a virtualized one. I'm relying on the L4STeam Github
>>>>>>>> project for the deployment of the L4S architecture and I am
>>>>>> looking
>>>>>>>> for someone who's familiar with the project and might be willing
>>>>>> to
>>>>>>>> help me: in fact I seem not to be able to achieve the correct
>>>>>>>> configuration.
>>>>>>>>>>> My setup is very simple: I have four servers (two senders and
>>>>>>>> two receivers) exchanging two traffic flows through one server
>>>>>>>> acting as a router. One client-server pair uses Prague as CC,
>>>>>> while
>>>>>>>> the other uses Cubic. All servers have the patched kernel
>>>>>> provided
>>>>>>>> in the https://github.com/L4STeam/linux/ repository branch.
>>>>>>>>>>> If I trigger a congestion on the router by generating both 
>>>>>>>>>>> the
>>>>>>>> Prague and the Cubic flows (let's say the flows measure 100
>>>>>> Mbit/s
>>>>>>>> each, and they come though a L2 switch both on the same router's
>>>>>>>> input interface on a 1Gb Ethernet link; only a 100M link though
>>>>>> is
>>>>>>>> in place on the output interface towards the receivers) I see 
>>>>>>>> the
>>>>>>>> L4S flow having higher delay, higher jitter and a smaller (and
>>>>>> more
>>>>>>>> variable) bandwidth share. The Prague share is 1/4 of the Cubic
>>>>>>>> share. I am sending an attachment with a graphical 
>>>>>>>> representation
>>>>>> of
>>>>>>>> the scenario here described.
>>>>>>>>>>> I configured my L4S endpoints as follows:
>>>>>>>>>>> - I set the CC as tcp Prague (sysctl -w
>>>>>>>> net.ipv4.tcp_congestion_control=prague)
>>>>>>>>>>> - I set the AccEcn, even if it's not necessary apparently
>>>>>>>> (sysctl -w net.ipv4.tcp_ecn=3)
>>>>>>>>>>> - I disabled the required offloading capabilities on the
>>>>>>>> endpoints (sudo ethtool -K $NETIF tso off gso off gro off lro
>>>>>> off)
>>>>>>>>>> [SM] I think you need to do the same on the router... or
>>>>>>>> with your
>>>>>>>>>> topology with running prague and cubic over separate 
>>>>>>>>>> end-points
>>>>>>>>>> especially on the router itself. Side-node, sch_cake grew a
>>>>>>>> split-gso
>>>>>>>>>> mode to automatically handle this issue because it can be a 
>>>>>>>>>> bit
>>>>>>>> of a
>>>>>>>>>> whack-a-mole problem to make these configs stick (and in the
>>>>>> case
>>>>>>>> of
>>>>>>>>>> cake the idea was to make deployment easy even for
>>>>>> non-experts).
>>>>>>>>> [MG] I tried as you suggested and unfortunately the situation
>>>>>>>> remains unvaried.
>>>>>>>> [SM2] Hmmm, that would indicate that it might not be
>>>>>>>> "lumpyness" of inputs into the router. I guess I would take
>>>>>> packet
>>>>>>>> captures on both interfaces of the router to see whether there 
>>>>>>>> is
>>>>>>>> any unexpected distribution of packets between both input and
>>>>>>>> output? Also worth looking is the CPU usage on the router... we
>>>>>>>> occasionally run into issues with aggressive?
>>>>>>>> power/voltage/frequency scaling where a CPU might take much
>>>>>> longer
>>>>>>>> to wake up than expected, the L-queue with its rather low (IMHO
>>>>>> too
>>>>>>>> low) reference delay of 1ms would be especially sensitive to 
>>>>>>>> such
>>>>>>>> issues.
>>>>>>>> Also does your 100Mbps interface support BQL?
>>>>>>>>> Still, I think I missed the point regarding sch_cake, could you
>>>>>>>> explain again what it is and if and how could it be useful?
>>>>>>>> [SM2] I am talking about Linux's cake qdisc and just as
>>>>>>>> example, cake does not support special treatment of ECT(1) but
>>>>>>>> implements rfc3168 ECN signaling for both ECT(0) and ECT(1). So
>>>>>> for
>>>>>>>> your experiments it might not be that useful (but for the fun of
>>>>>> it,
>>>>>>>> maybe try it as alternative for DualQ) I just mentioned it as an
>>>>>>>> example for a qdisc that opted for not simply disabling all
>>>>>>>> offloads. After all these offloads are quite useful, as they can
>>>>>>>> considerably reduce the CPU of networking. (GSO/GRO work by
>>>>>>>> ameliorating the somewhat fixed per-packet cost of Linux
>>>>>>>> network-stack over multiple ethernet frames, as long as the
>>>>>>>> increased deelay inherent in such bathing approaches this can
>>>>>> help a
>>>>>>>> lot).
>>>>>>>>> Apologize, I guess I perfectly fit into the definition of "non
>>>>>>>> experts". I tried to look it up on the internet but I struggled
>>>>>> to
>>>>>>>> find any clarification.
>>>>>>>> [SM2] Sorry, my bad, I should have been clearer that I was
>>>>>>>> talkning about a qdisc here, see "man tc-cake" on a sufficietly
>>>>>>>> modern Linux system, the source code file is called sch_cake.c
>>>>>> (see
>>>>>>>> e.g.
>>>>>> https://elixir.bootlin.com/linux/latest/source/net/sched/sch_cake.c)
>>>>>>>>>>> - I configured the fair queue on the endpoints (sudo tc qdisc
>>>>>>>> replace dev $NETIF root fq)
>>>>>>>>>>> I configured my router as follows:
>>>>>>>>>>> - I enabled forwarding through these interfaces to obtain the
>>>>>>>> routing capabilities (sudo sysctl -w net.ipv4.ip_forward=1)
>>>>>>>>>>> - I set the dualpi2 on both interfaces (sudo tc qdisc replace
>>>>>>>> dev $NETIF root dualpi2)
>>>>>>>>>>> I then applied the fair queue and disabled the offloading
>>>>>>>> capabilities on both my classic endpoints to ensure that the
>>>>>> classic
>>>>>>>> and l4s flows act as fairly as possible, but to no avail (even
>>>>>>>> without these precautions the results remain roughly the same).
>>>>>>>>>> [SM] Again, I think with your topology offloads at the
>>>>>>>> endpoints
>>>>>>>>>> should not have much influence, but at the router the well
>>>>>> might.
>>>>>>>> If
>>>>>>>>>> that turns out to help this might be explained by Prague's
>>>>>>>> (and/or
>>>>>>>>>> DualQ's L-queue) considerably higher sensitivity to bursty
>>>>>>>> traffic
>>>>>>>>>> compared to classic traffic and queue.
>>>>>>>>>>> I am sure I am missing some important details in the setup,
>>>>>> and
>>>>>>>> I would really appreciate some help.
>>>>>>>>>> [SM] To me this looks rather straight forward, and I
>>>>>>>> probably would
>>>>>>>>>> try something similar, but I did not actually try in practice.
>>>>>>>>>> Regards & good luck
>>>>>>>>>> Sebastian
>>>>>>>>> [MG] Thanks in advance for your help, and if you have other
>>>>>>>> tips or if you (or anyone else for that matter) are by any 
>>>>>>>> chance
>>>>>>>> aware of a paper or project using the prague branch of the
>>>>>> L4STeam
>>>>>>>> repository, that might indeed be really helpful too.
>>>>>>>> [SM] I am not the best/most objective person to quizz here,
>>>>>>>> as I consider L4S in general too little too late and neither TCP
>>>>>>>> Prague nor the DualQ AQM worth deploying in their current state
>>>>>> (but
>>>>>>>> that is why I consider your effort researching these admirable,
>>>>>> both
>>>>>>>> IMHO really need more research direly).
>>>>>>>> I would always try to run the same tests over a bottleneck using
>>>>>> a
>>>>>>>> fq-scheduler, be it the all in one cake or fq_codel. Fq_codel
>>>>>>>> actually con be configured to treat ECT(1) mire in line with 
>>>>>>>> what
>>>>>>>> TCP Prague desires, so that might well be a decent starting 
>>>>>>>> point
>>>>>>>> for alternative measurements....
>>>>>>>> Regards
>>>>>>>> Sebastian
>>>>>>>>> My best regards to you and the community,
>>>>>>>>> Matteo
>>>>>>>>>>> Regards,
>>>>>>>>>>> Matteo
>>>>>>>>>>> P.s.
>>>>>>>>>>> I just want to point out that by looking at the packet traces
>>>>>>>> everything seems fine: Prague carries the ECN=1, the dualpi2
>>>>>> marks
>>>>>>>> packets with ECN=3, the AccEcn control signals on the ACE fields
>>>>>> are
>>>>>>>> coherent, and no losses occur in the Prague flow, while they do
>>>>>>>> happen with the Cubic flow. It looks like Prague is
>>>>>> underperforming
>>>>>>>> for whatever reason. Furthermore, if I switch back to two Cubic
>>>>>>>> flows I measure perfect share, equal delay and equal jitter, so
>>>>>> it
>>>>>>>> looks to me like there are no physical impairments on the
>>>>>>>> testbed.<testplant_issue.pdf>--
>>>>>>>>>>> L4s-discuss mailing list
>>>>>>>>>>> [e-mail]
>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>>>>> --
>>>>>>>>> L4s-discuss mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>>>>>> --
>>>>>>>> L4s-discuss mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss--
>>>>>> L4s-discuss mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>>>> --
>>>> L4s-discuss mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/l4s-discuss
>> 
>> --
>> L4s-discuss mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/l4s-discuss