Re: [rmcat] Updates for Test Case 5.5 and 5.6 in draft-ietf-rmcat-eval-test

[Michael Welzl <[e-mail]>]

> On 4. sep. 2015, at 22.18, Xiaoqing Zhu (xiaoqzhu) <[e-mail]> wrote:
> 
> Hi Michael, 
> 
> Thanks for your input.  A few more comments below starting with [XZ]
> 
> Best,
> Xiaoqing
> 
> 
> On 9/4/15, 2:21 AM, "Michael Welzl" <[e-mail]> wrote:
> 
>> 
>>> On 03 Sep 2015, at 22:22, Xiaoqing Zhu (xiaoqzhu) <[e-mail]>
>>> wrote:
>>> 
>>> Hi all, 
>>> 
>>> We are about to update the test case draft to keep it alive by
>>> September 11, 2015. And, as co-author, I would like to poll the mailing
>>> list regarding two proposed changes for Test Case 5.5 and 5.6.
>>> 	€ Test Case 5.5 (RTT Fairness):  currently all flows start gradually,
>>> and that may lead to late-coming flows settling at higher rates if they
>>> rely on queuing delay measurements.  Since the intention of this test
>>> case is to measure fairness w.r.t RTT, would it make more sense to start
>>> all flows at the same time, or roughly the same time (e.g., random start
>>> within 5 seconds) to avoid flow synchronization effects?  Notice that if
>>> any candidate scheme has an issue with late-comer advantage, that should
>>> have already been exposed by Test Case 5.4.
>> 
>> I'd say no to this one: I don't understand how starting at the same time
>> or roughly at the same time would avoid flow synchronization effects?
>> RTT Fairness should clearly work if flows join later or all at roughly
>> the same time. Also note that randomness only makes sense to introduce if
>> there is a minimum number of repetitions and we consider the distribution
>> of output data, i.e. it's not particularly useful when showing plots over
>> time as is the common thing here.
> 
> 
> [XZ] Agree that RTT fairness certainly works even if flows gradually
> start. Just that instead of observing same rate per flow (regardless of
> RTT), we would observe later flows at higher rate due to late-comer
> advantage.  Basically the fairness evaluation will be ³muddled² by a
> separate factor independent of path RTT.
> 
> [XZ] I like your comment on how experiments involving randomness should be
> evaluated.  So, maybe a different approach is to repeat the test case but
> with random ordering of the flow start times multiple times and compare
> the average rate per flow.  Then we don¹t need to start all flows at the
> same time. The change to the description of test case will be minimal.
> Will people like this approach better than starting all flows at same
> time?    

Hm, I didn’t suggest to do this, I just tried to explain why I think randomness is not a good idea to add.
This sounds to me like making it more complicated. If you introduce random ordering of flow start times, you’ll need to make a choice for the interval over which randomness applies. On what basis? If it gets large, you can be back at flows gradually starting again.

All in all, I’m not sure I agree if it makes sense to say that flows gradually starting “muddles” the behavior - you could also say that starting at the same time is a special case…


>>> 	€ Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue
>>> sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes
>>> (CoDel and PIE) are striving to contain queuing delay with a target of
>>> 20ms, while allowing for a larger physical queue size to accommodate
>>> instantaneous bursts, a value of 20m for a drop-tail queue depth seems
>>> rather out of place.  Should we simply remove the value of 20ms for now?
>>> If we want testers to play with a lower queue depth, we may start with
>>> 100ms of queuing, which corresponds to the bandwidth-delay product along
>>> that path (50ms propagation each direction).  On a related note: the
>>> queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is
>>> specified as 70ms. Just wondering how that number came about?
>> 
>> I have no strong opinion on this one; though, in a way I'd say it's a
>> good thing that the 20ms case highlights what happens when the queue is
>> below the target, we should at least know what controllers do under such
>> circumstances?  But then this may just twist the result strangely if it's
>> part of a test case, not a test case on its own, so I can understand that
>> it seems out of place, as you say...
> 
> 
> [XZ] Not sure if I understand what you mean by ³when the queue is below
> the target²?

I meant, what if e.g. you have a 10ms queue but congestion control mechanisms try to operate at 20ms?
(20 / 20 is a special case at the borderline between this (“below the target”) and “normal”)


> How would source congestion control behave if we indeed have
> a bottleneck queue depth of 20ms? I¹d say people are welcome to test out
> that scenario. Just that the setup itself does not seem to be motivated by
> a practical deployment scenario.
> 
> [XZ] In any case I suppose we share the same viewpoint on this one.

Yes

Cheers,
Michael