[tsvwg] sce vs l4s comparison plots?

[Dave Taht <[e-mail]>]

It is really wonderful to have these two groups using the
flent tool and producing comparable data on the same test suites in
a few cases now, and the testing so far found a couple bugs.

https://github.com/heistp/sce-l4s-bakeoff
https://l4s.cablelabs.com/

I do feel rather strongly that any major change to how the internet does
congestion control requires a maximum of publically repeatable
benchmarks using open, shared source code. [1]

I do keep hoping for a rrul test on a 15 to 20x1 asymmetric network
simulation like how most cablemodems and dsl boxes are configured, and
more of the rtt_fair_* tests, and what the heck - the tcp_square_wave
test, and any of a dozen others. tcp_nup with 32 flows... heck, hundreds
of flows... go to town! Burn some cycles!

A few packet caps would be nice to have also, to look
over relative mark rates in tcptrace and scetrace.

Anyway... l4steam....

In looking over the l4s.cablelabs site today though, I see the data is
only available as .png files, and that the associated flent.gz data
is not apparently being published.

.svgs help. Flent can also be run at a higher sample rate... but

There are a few things about having *.flent.gz files available that
would help us look at some things in more detail. Me, I tend to look at
comparison cdf plots more than anything else, and secondly bar plots,
and lacking that it's hard to make a distinction betwen a whole bunch of
tiny lines at the bottom about overall queue depths and latency
distributions, among other things, without squinting, and doing tricks
in photoshop to make the two or more datasets "line up".

(there's dozens of plot types in the flent tool, and comparison plots
are a snap)

By default (when run without -x) flent captures very little metadata
about the system it is run on (IP addresses, a couple sysctls, and
qdiscs) but it's helpful to have. One example that would be in that
metadata, is that I'm unsure if the ns3 data is using an IW4 or IW10?

It sounds like you are rate limiting with htb? (to what quantum?)

Another example, in more "native" environments running at a simulated
line rate, BQL is quite important to have in the simulation
also. there's been a couple papers published on BQL's benefits vs a raw
txring, thus far, there's a good plot of what it does in fig 6 of:

http://sci-hub.tw/10.1109/LANMAN.2019.8847054

Lastly...

So far as I know ns(X) does not correctly simulate GSO/TSO even when
run in DCE mode, but I could be out of date on that. TBF (and cake)
do break apart superpackets, htb (+ anything, like fq_codel or dualq)
do not.

I also have completely forgotten at this point, to what extent the
ns3 model of fq_codel matches the deployed reality. It was such a long
time ago.... as best as I recall ns3 didn't even have ecn support then!

I'm taking apart
https://gitlab.com/tomhend/modules/l4s-evaluation/blob/master/README.md
as I write.

thx for the data!

[1] If anyone hasn't read my related rant yet, it's here:
https://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf