Re: [rmcat] Updates for Test Case 5.5 and 5.6 in draft-ietf-rmcat-eval-test

["Xiaoqing Zhu (xiaoqzhu)" <[e-mail]>]

Hi Michael, 

Thanks for your input.  A few more comments below starting with [XZ]

Best,
Xiaoqing


On 9/4/15, 2:21 AM, "Michael Welzl" <[e-mail]> wrote:

>
>> On 03 Sep 2015, at 22:22, Xiaoqing Zhu (xiaoqzhu) <[e-mail]>
>>wrote:
>> 
>> Hi all, 
>> 
>> We are about to update the test case draft to keep it alive by
>>September 11, 2015. And, as co-author, I would like to poll the mailing
>>list regarding two proposed changes for Test Case 5.5 and 5.6.
>> 	€ Test Case 5.5 (RTT Fairness):  currently all flows start gradually,
>>and that may lead to late-coming flows settling at higher rates if they
>>rely on queuing delay measurements.  Since the intention of this test
>>case is to measure fairness w.r.t RTT, would it make more sense to start
>>all flows at the same time, or roughly the same time (e.g., random start
>>within 5 seconds) to avoid flow synchronization effects?  Notice that if
>>any candidate scheme has an issue with late-comer advantage, that should
>>have already been exposed by Test Case 5.4.
>
>I'd say no to this one: I don't understand how starting at the same time
>or roughly at the same time would avoid flow synchronization effects?
>RTT Fairness should clearly work if flows join later or all at roughly
>the same time. Also note that randomness only makes sense to introduce if
>there is a minimum number of repetitions and we consider the distribution
>of output data, i.e. it's not particularly useful when showing plots over
>time as is the common thing here.


[XZ] Agree that RTT fairness certainly works even if flows gradually
start. Just that instead of observing same rate per flow (regardless of
RTT), we would observe later flows at higher rate due to late-comer
advantage.  Basically the fairness evaluation will be ³muddled² by a
separate factor independent of path RTT.

[XZ] I like your comment on how experiments involving randomness should be
evaluated.  So, maybe a different approach is to repeat the test case but
with random ordering of the flow start times multiple times and compare
the average rate per flow.  Then we don¹t need to start all flows at the
same time. The change to the description of test case will be minimal.
Will people like this approach better than starting all flows at same
time?    


>
>
>> 	€ Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue
>>sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes
>>(CoDel and PIE) are striving to contain queuing delay with a target of
>>20ms, while allowing for a larger physical queue size to accommodate
>>instantaneous bursts, a value of 20m for a drop-tail queue depth seems
>>rather out of place.  Should we simply remove the value of 20ms for now?
>>  If we want testers to play with a lower queue depth, we may start with
>>100ms of queuing, which corresponds to the bandwidth-delay product along
>>that path (50ms propagation each direction).  On a related note: the
>>queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is
>>specified as 70ms. Just wondering how that number came about?
>
>I have no strong opinion on this one; though, in a way I'd say it's a
>good thing that the 20ms case highlights what happens when the queue is
>below the target, we should at least know what controllers do under such
>circumstances?  But then this may just twist the result strangely if it's
>part of a test case, not a test case on its own, so I can understand that
>it seems out of place, as you say...


[XZ] Not sure if I understand what you mean by ³when the queue is below
the target²? How would source congestion control behave if we indeed have
a bottleneck queue depth of 20ms? I¹d say people are welcome to test out
that scenario. Just that the setup itself does not seem to be motivated by
a practical deployment scenario.

[XZ] In any case I suppose we share the same viewpoint on this one.



>
>
>Cheers,
>Michael
>