Re: [tsvwg] [tcpm] sce vs l4s comparison plots?

"Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net> Sun, 10 November 2019 03:31 UTC

Return-Path: <4bone@gndrsh.dnsmgr.net>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E361F1200FB; Sat, 9 Nov 2019 19:31:27 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.497
X-Spam-Level:
X-Spam-Status: No, score=-1.497 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.4, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7TZD8-1krbEI; Sat, 9 Nov 2019 19:31:26 -0800 (PST)
Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E87DF1200F6; Sat, 9 Nov 2019 19:31:25 -0800 (PST)
Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xAA3VNgZ027400; Sat, 9 Nov 2019 19:31:23 -0800 (PST) (envelope-from 4bone@gndrsh.dnsmgr.net)
Received: (from 4bone@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xAA3VNmp027399; Sat, 9 Nov 2019 19:31:23 -0800 (PST) (envelope-from 4bone)
From: "Rodney W. Grimes" <4bone@gndrsh.dnsmgr.net>
Message-Id: <201911100331.xAA3VNmp027399@gndrsh.dnsmgr.net>
In-Reply-To: <87mud4ejl9.fsf@taht.net>
To: Dave Taht <dave@taht.net>
Date: Sat, 9 Nov 2019 19:31:23 -0800 (PST)
CC: tsvwg IETF list <tsvwg@ietf.org>, tcpm@ietf.org
X-Mailer: ELM [version 2.4ME+ PL121h (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/vz7oytwKF48yHwBFQMkqAW4elMc>
Subject: Re: [tsvwg] [tcpm] sce vs l4s comparison plots?
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 10 Nov 2019 03:31:28 -0000

Dave,
	Thank you for the links, some short responses inline.

Regards,
Rod

> 
> It is really wonderful to have these two groups using the
> flent tool and producing comparable data on the same test suites in
> a few cases now, and the testing so far found a couple bugs.
> 
> https://github.com/heistp/sce-l4s-bakeoff
> https://l4s.cablelabs.com/
> 
> I do feel rather strongly that any major change to how the internet does
> congestion control requires a maximum of publically repeatable
> benchmarks using open, shared source code. [1]
> 
> I do keep hoping for a rrul test on a 15 to 20x1 asymmetric network
> simulation like how most cablemodems and dsl boxes are configured, and
> more of the rtt_fair_* tests, and what the heck - the tcp_square_wave
> test, and any of a dozen others. tcp_nup with 32 flows... heck, hundreds
> of flows... go to town! Burn some cycles!

I believe Pete has some work scheduled for the (near?) future
in this area.

> 
> A few packet caps would be nice to have also, to look
> over relative mark rates in tcptrace and scetrace.
I do believe all you seek for Petes data is there,
https://github.com/heistp/sce-l4s-bakeoff/#test-output
specifically items:
*.flent.gz- standard Flent output file, may be used to view original data or re-generate plots
*.tcpdump_*.pcap.bz2- pcap file compressed with bzip2 -9

> 
> Anyway... l4steam....
> 
> In looking over the l4s.cablelabs site today though, I see the data is
> only available as .png files, and that the associated flent.gz data
> is not apparently being published.
> 
> ..svgs help. Flent can also be run at a higher sample rate... but
> 
> There are a few things about having *.flent.gz files available that
> would help us look at some things in more detail. Me, I tend to look at
> comparison cdf plots more than anything else, and secondly bar plots,
> and lacking that it's hard to make a distinction betwen a whole bunch of
> tiny lines at the bottom about overall queue depths and latency
> distributions, among other things, without squinting, and doing tricks
> in photoshop to make the two or more datasets "line up".
> 
> (there's dozens of plot types in the flent tool, and comparison plots
> are a snap)
> 
> By default (when run without -x) flent captures very little metadata
> about the system it is run on (IP addresses, a couple sysctls, and
> qdiscs) but it's helpful to have. One example that would be in that
> metadata, is that I'm unsure if the ns3 data is using an IW4 or IW10?
> 
> It sounds like you are rate limiting with htb? (to what quantum?)
> 
> Another example, in more "native" environments running at a simulated
> line rate, BQL is quite important to have in the simulation
> also. there's been a couple papers published on BQL's benefits vs a raw
> txring, thus far, there's a good plot of what it does in fig 6 of:
> 
> http://sci-hub.tw/10.1109/LANMAN.2019.8847054
> 
> Lastly...
> 
> So far as I know ns(X) does not correctly simulate GSO/TSO even when
> run in DCE mode, but I could be out of date on that. TBF (and cake)
> do break apart superpackets, htb (+ anything, like fq_codel or dualq)
> do not.
> 
> I also have completely forgotten at this point, to what extent the
> ns3 model of fq_codel matches the deployed reality. It was such a long
> time ago.... as best as I recall ns3 didn't even have ecn support then!

And apparently is still missing some ECN (RFC3168) support as your
following link contains some patches:
	6.  cd back one level to the top-level ns-3 directory,
	and patch the ns-3.30.1 release with a patch provided
	in the l4s-evaluation/patches/ directory. This patch
	adds support for ECE handling and DCTCP in the base
	TCP code, and adds ECN handling to CoDel and FqCoDel models.

> 
> I'm taking apart
> https://gitlab.com/tomhend/modules/l4s-evaluation/blob/master/README.md
> as I write.
> 
> thx for the data!
> 
> [1] If anyone hasn't read my related rant yet, it's here:
> https://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf
> 
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
> 

-- 
Rod Grimes                                                 rgrimes@freebsd.org