Re: [tsvwg] ECN encapsulation draft - proposed resolution

[Bob Briscoe <[e-mail]>]

Jonathan,

On 22/06/2021 20:38, Jonathan Morton wrote:
>> On 22 Jun, 2021, at 5:02 pm, Markku Kojo <[e-mail]> wrote:
> Up front, let me say that I mainly agree with your approach to this.
>
>> And, if the number of flows is increased, then the time between the marks decreases. So, we need to be careful in understanding what we are modelling. Otherwise, if we just play with the formulae, the outcome is just crap.
> So, rather than playing with formulae, I organised an empirical test over the weekend to see what the effects really are.  That accounts for the long pause before replying.
>
> 	https://sce.dnsmgr.net/results/mss-tests/
>
> The test involves three flows with the same CC algorithm running through a common single-queue, single-instance AQM.  The AQM is Codel as an exemplar of time-domain marking, or PIE as an exemplar of packet-mode marking, both with ECN enabled.  The test was repeated for Reno and CUBIC, as exemplars of standards-track CC, and DCTCP as an exemplar of 1/p response.  The path parameters are 50Mbps throughput, 80ms RTT, sufficient to keep CUBIC out of "Reno compatibility" mode.
>
> The three flows differ in terms of packet and segment size.  Flow 1 is a conventional bulk flow using a full 1460-byte MSS.  Flow 2 reduces this to 730-byte MSS.  Flow 3 uses a 1460-byte MSS, but goes through a path with reduced MTU (with PMTUD disabled) causing fragmentation into two packets per segment, so has the same packet rate at the AQM as Flow 2.

[BB] Can you quantify the MTU? So we know the sizes of the first and 
second of each pair of fragments. I see a legend label later with 1280 
in it. Is that the MTU? It doesn't actually matter to the main outcome, 
I'd just like to be able to double check.

> Fragmentation reassembly is as per RFC-3168.
>
> A well-conditioned congestion system should give each flow roughly equal application goodput on average.  Some allowance can be made for bandwidth lost to headers in the smaller packets, but this is a small effect.  But what do we actually get?

[BB] In /this/ exercise, we are not trying to make flows with different 
packet sizes have the same bit-rate, if they didn't have the same 
bit-rate without fragmentation.

While that might be an interesting research project, it is boiling the 
ocean compared to the task in hand. We are only trying to ensure that 
flows that are reframed at a lower layer, then reassembled, end up with 
the same rate as if they had not been reframed  then reassembled (modulo 
any slight differences due to differences in total header sizes, as you 
say). That is the art of designing an experiment that only changes the 
one parameter of interest.


> The theoretically ideal case is DCTCP with time-domain marking, where the expected goodput is exactly equal for all three flows.  The actual observed goodput is not quite equal, but reasonably close and very consistent over a 10-minute period:
>
> 	https://sce.dnsmgr.net/results/mss-tests/dctcp-fq_codel-plot.html

[BB] You started out the email saying CoDel. However, the title of this 
plot (and the file name) says fq_codel. Surely you didn't mean to use FQ 
for this experiment? If the bottleneck includes a /scheduler/ designed 
to give each flow the same bit-rate, it will do that - irrespective of 
packet sizes, fragmentation, framing, or anything else for that matter.

>
> But if we apply packet-mode marking, the smaller packets are markedly disadvantaged compared to larger ones, approximately in proportion to the average packet size, and again this is consistent for 10 minutes straight:
>
> 	https://sce.dnsmgr.net/results/mss-tests/dctcp-pie-plot.html

[BB] This proves that the disadvantage of using small packets has 
nothing to do with fragmentation or reassembly, because both flows 1 & 2 
do not go through fragmentation/reassembly, and flow 2 has half the rate 
of flow 1, because its packets are half the size. This is the well-known 
fact that flows using smaller packets disadvantage themselves. That has 
nothing to do with fragmentation or reassembly.

In fact, it shows exactly what I predicted. Even though flow 3 sends the 
same size packets as flow 1 and is identical in every way, it ends up 
with half the rate, the only difference having been that flow 3 goes 
through RFC3168 fragmentation and reassembly with the AQM marking ECN in 
between.


>
> I do not have running code with which to test a "marked bytes preserving" reassembly rule.  However, such a rule could only possibly influence Flow 3, as that is the only one that's fragmented.  In theory it would be elevated to the level of Flow 1, leaving Flow 2 as the only disadvantaged flow.
>
> You might say that using small MSS is not reasonable for capacity-seeking traffic.  But recall what Paul Vixie pointed out in the other thread about UDP Options, that the MTU over Internet paths could reasonably increase in future.  This implies that "jumbo" TCP segments would, at least for some time, share space with traffic still using today's Ethernet-sized packets, and the latter would be in the analogous position of Flow 2 in this test.  So do not dismiss Flow 2 as irrelevant to your interests.
>
> Moving on to Reno, we see roughly the same effect, though the large Reno sawtooth means we have to peer through a lot of noise to see the trends.  Nevertheless they are similar to DCTCP over a long average:
>
> 	https://sce.dnsmgr.net/results/mss-tests/reno-fq_codel-plot.html
> 	https://sce.dnsmgr.net/results/mss-tests/reno-pie-plot.html
>
> And further likewise for CUBIC:
>
> 	https://sce.dnsmgr.net/results/mss-tests/cubic-fq_codel-plot.html
> 	https://sce.dnsmgr.net/results/mss-tests/cubic-pie-plot.html
>
> These results with standards-track CC confirm those from DCTCP, and the same logic applies.
>
> Current practice on the Internet is to use time-domain ECN marking (because Codel is the most widely deployed AQM with ECN enabled) and RFC-3168 fragment reassembly.  The above results show that this is clearly superior to packet-mode marking with either RFC-3168 or "byte-preserving" reassembly rules.  Hence we should probably try to avoid encouraging the latter behaviour in new Internet specifications.

As above, the PIE results prove exactly what I predicted. And the 
FQ_CoDel results don't say anything about the effect of fragmentation 
and reassembly on flow rate, because flow rate is enforced by the scheduler.

Now, it will be interesting to see what happens if the AQM is CoDel 
rather than FQ-CoDel.


Cheers



Bob

>   - Jonathan Morton
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/