Re: [tsvwg] ECN encapsulation draft - proposed resolution

[Jonathan Morton <[e-mail]>]

> On 22 Jun, 2021, at 5:02 pm, Markku Kojo <[e-mail]> wrote:

Up front, let me say that I mainly agree with your approach to this.

> And, if the number of flows is increased, then the time between the marks decreases. So, we need to be careful in understanding what we are modelling. Otherwise, if we just play with the formulae, the outcome is just crap.

So, rather than playing with formulae, I organised an empirical test over the weekend to see what the effects really are.  That accounts for the long pause before replying.

	https://sce.dnsmgr.net/results/mss-tests/

The test involves three flows with the same CC algorithm running through a common single-queue, single-instance AQM.  The AQM is Codel as an exemplar of time-domain marking, or PIE as an exemplar of packet-mode marking, both with ECN enabled.  The test was repeated for Reno and CUBIC, as exemplars of standards-track CC, and DCTCP as an exemplar of 1/p response.  The path parameters are 50Mbps throughput, 80ms RTT, sufficient to keep CUBIC out of "Reno compatibility" mode.

The three flows differ in terms of packet and segment size.  Flow 1 is a conventional bulk flow using a full 1460-byte MSS.  Flow 2 reduces this to 730-byte MSS.  Flow 3 uses a 1460-byte MSS, but goes through a path with reduced MTU (with PMTUD disabled) causing fragmentation into two packets per segment, so has the same packet rate at the AQM as Flow 2.  Fragmentation reassembly is as per RFC-3168.

A well-conditioned congestion system should give each flow roughly equal application goodput on average.  Some allowance can be made for bandwidth lost to headers in the smaller packets, but this is a small effect.  But what do we actually get?

The theoretically ideal case is DCTCP with time-domain marking, where the expected goodput is exactly equal for all three flows.  The actual observed goodput is not quite equal, but reasonably close and very consistent over a 10-minute period:

	https://sce.dnsmgr.net/results/mss-tests/dctcp-fq_codel-plot.html

But if we apply packet-mode marking, the smaller packets are markedly disadvantaged compared to larger ones, approximately in proportion to the average packet size, and again this is consistent for 10 minutes straight:

	https://sce.dnsmgr.net/results/mss-tests/dctcp-pie-plot.html

I do not have running code with which to test a "marked bytes preserving" reassembly rule.  However, such a rule could only possibly influence Flow 3, as that is the only one that's fragmented.  In theory it would be elevated to the level of Flow 1, leaving Flow 2 as the only disadvantaged flow.

You might say that using small MSS is not reasonable for capacity-seeking traffic.  But recall what Paul Vixie pointed out in the other thread about UDP Options, that the MTU over Internet paths could reasonably increase in future.  This implies that "jumbo" TCP segments would, at least for some time, share space with traffic still using today's Ethernet-sized packets, and the latter would be in the analogous position of Flow 2 in this test.  So do not dismiss Flow 2 as irrelevant to your interests.

Moving on to Reno, we see roughly the same effect, though the large Reno sawtooth means we have to peer through a lot of noise to see the trends.  Nevertheless they are similar to DCTCP over a long average:

	https://sce.dnsmgr.net/results/mss-tests/reno-fq_codel-plot.html
	https://sce.dnsmgr.net/results/mss-tests/reno-pie-plot.html

And further likewise for CUBIC:

	https://sce.dnsmgr.net/results/mss-tests/cubic-fq_codel-plot.html
	https://sce.dnsmgr.net/results/mss-tests/cubic-pie-plot.html

These results with standards-track CC confirm those from DCTCP, and the same logic applies.

Current practice on the Internet is to use time-domain ECN marking (because Codel is the most widely deployed AQM with ECN enabled) and RFC-3168 fragment reassembly.  The above results show that this is clearly superior to packet-mode marking with either RFC-3168 or "byte-preserving" reassembly rules.  Hence we should probably try to avoid encouraging the latter behaviour in new Internet specifications.

 - Jonathan Morton