Re: [tsvwg] ECN encapsulation draft - proposed resolution

[Jonathan Morton <[e-mail]>]

>> So, rather than playing with formulae, I organised an empirical test over the weekend to see what the effects really are.  That accounts for the long pause before replying.
>> 
>> 	https://sce.dnsmgr.net/results/mss-tests/
>> 
>> The test involves three flows with the same CC algorithm running through a common single-queue, single-instance AQM.  The AQM is Codel as an exemplar of time-domain marking, or PIE as an exemplar of packet-mode marking, both with ECN enabled.  The test was repeated for Reno and CUBIC, as exemplars of standards-track CC, and DCTCP as an exemplar of 1/p response.  The path parameters are 50Mbps throughput, 80ms RTT, sufficient to keep CUBIC out of "Reno compatibility" mode.
>> 
>> The three flows differ in terms of packet and segment size.  Flow 1 is a conventional bulk flow using a full 1460-byte MSS.  Flow 2 reduces this to 730-byte MSS.  Flow 3 uses a 1460-byte MSS, but goes through a path with reduced MTU (with PMTUD disabled) causing fragmentation into two packets per segment, so has the same packet rate at the AQM as Flow 2.
> 
> [BB] Can you quantify the MTU? So we know the sizes of the first and second of each pair of fragments. I see a legend label later with 1280 in it. Is that the MTU? It doesn't actually matter to the main outcome, I'd just like to be able to double check.

Agree that the series labels aren't as clear as they should be.  The third flow uses an MSS of 1460 (same as the first flow), but is sent through a link with an MTU of 1280, with MTU discovery disabled so that it gets fragmented there.  The second flow uses a smaller MSS and is not fragmented.

>> Fragmentation reassembly is as per RFC-3168.
>> 
>> A well-conditioned congestion system should give each flow roughly equal application goodput on average.  Some allowance can be made for bandwidth lost to headers in the smaller packets, but this is a small effect.  But what do we actually get?
> 
> [BB] In /this/ exercise, we are not trying to make flows with different packet sizes have the same bit-rate, if they didn't have the same bit-rate without fragmentation.
> 
> While that might be an interesting research project, it is boiling the ocean compared to the task in hand. We are only trying to ensure that flows that are reframed at a lower layer, then reassembled, end up with the same rate as if they had not been reframed  then reassembled (modulo any slight differences due to differences in total header sizes, as you say). That is the art of designing an experiment that only changes the one parameter of interest.

When we already have a simple solution that also solves the larger problem, why not just use it?

>> The theoretically ideal case is DCTCP with time-domain marking, where the expected goodput is exactly equal for all three flows.  The actual observed goodput is not quite equal, but reasonably close and very consistent over a 10-minute period:
>> 
>> 	https://sce.dnsmgr.net/results/mss-tests/dctcp-fq_codel-plot.html
> 
> [BB] You started out the email saying CoDel. However, the title of this plot (and the file name) says fq_codel. Surely you didn't mean to use FQ for this experiment? If the bottleneck includes a /scheduler/ designed to give each flow the same bit-rate, it will do that - irrespective of packet sizes, fragmentation, framing, or anything else for that matter.

If you look more carefully, you'll notice that we used fq_codel with the "flows 1" argument, which disables the FQ mechanism and places all traffic in the same FIFO and the same set of AQM state.  So we accurately describe it in prose as just "Codel".  We could just as validly have used Cake with the "besteffort flowblind" arguments to achieve a similar effect.

>> But if we apply packet-mode marking, the smaller packets are markedly disadvantaged compared to larger ones, approximately in proportion to the average packet size, and again this is consistent for 10 minutes straight:
>> 
>> 	https://sce.dnsmgr.net/results/mss-tests/dctcp-pie-plot.html
> 
> [BB] This proves that the disadvantage of using small packets has nothing to do with fragmentation or reassembly, because both flows 1 & 2 do not go through fragmentation/reassembly, and flow 2 has half the rate of flow 1, because its packets are half the size. This is the well-known fact that flows using smaller packets disadvantage themselves. That has nothing to do with fragmentation or reassembly.
> 
> In fact, it shows exactly what I predicted. Even though flow 3 sends the same size packets as flow 1 and is identical in every way, it ends up with half the rate, the only difference having been that flow 3 goes through RFC3168 fragmentation and reassembly with the AQM marking ECN in between.

This is an accurate statement of the problem to be solved.  Time-domain marking (as used by Codel) combined with the existing RFC-3168 reassembly rule solves the problem.  Packet-domain marking exposes the problem on both flows 2 and 3, and your proposed reassembly rule solves it for only flow 3, not for flow 2.

 - Jonathan Morton