Re: [rmcat] Updates for Test Case 5.5 and 5.6 in draft-ietf-rmcat-eval-test

["Xiaoqing Zhu (xiaoqzhu)" <[e-mail]>]

Trying to clarify further with responses below inline.



On 9/4/15, 4:21 PM, "Michael Welzl" <[e-mail]> wrote:

>
>> On 4. sep. 2015, at 22.18, Xiaoqing Zhu (xiaoqzhu) <[e-mail]>
>>wrote:
>> 
>> Hi Michael, 
>> 
>> Thanks for your input.  A few more comments below starting with [XZ]
>> 
>> Best,
>> Xiaoqing
>> 
>> 
>> On 9/4/15, 2:21 AM, "Michael Welzl" <[e-mail]> wrote:
>> 
>>> 
>>>> On 03 Sep 2015, at 22:22, Xiaoqing Zhu (xiaoqzhu) <[e-mail]>
>>>> wrote:
>>>> 
>>>> Hi all, 
>>>> 
>>>> We are about to update the test case draft to keep it alive by
>>>> September 11, 2015. And, as co-author, I would like to poll the
>>>>mailing
>>>> list regarding two proposed changes for Test Case 5.5 and 5.6.
>>>> 	€ Test Case 5.5 (RTT Fairness):  currently all flows start gradually,
>>>> and that may lead to late-coming flows settling at higher rates if
>>>>they
>>>> rely on queuing delay measurements.  Since the intention of this test
>>>> case is to measure fairness w.r.t RTT, would it make more sense to
>>>>start
>>>> all flows at the same time, or roughly the same time (e.g., random
>>>>start
>>>> within 5 seconds) to avoid flow synchronization effects?  Notice that
>>>>if
>>>> any candidate scheme has an issue with late-comer advantage, that
>>>>should
>>>> have already been exposed by Test Case 5.4.
>>> 
>>> I'd say no to this one: I don't understand how starting at the same
>>>time
>>> or roughly at the same time would avoid flow synchronization effects?
>>> RTT Fairness should clearly work if flows join later or all at roughly
>>> the same time. Also note that randomness only makes sense to introduce
>>>if
>>> there is a minimum number of repetitions and we consider the
>>>distribution
>>> of output data, i.e. it's not particularly useful when showing plots
>>>over
>>> time as is the common thing here.
>> 
>> 
>> [XZ] Agree that RTT fairness certainly works even if flows gradually
>> start. Just that instead of observing same rate per flow (regardless of
>> RTT), we would observe later flows at higher rate due to late-comer
>> advantage.  Basically the fairness evaluation will be ³muddled² by a
>> separate factor independent of path RTT.
>> 
>> [XZ] I like your comment on how experiments involving randomness should
>>be
>> evaluated.  So, maybe a different approach is to repeat the test case
>>but
>> with random ordering of the flow start times multiple times and compare
>> the average rate per flow.  Then we don¹t need to start all flows at the
>> same time. The change to the description of test case will be minimal.
>> Will people like this approach better than starting all flows at same
>> time?    
>
>Hm, I didn’t suggest to do this, I just tried to explain why I think
>randomness is not a good idea to add.
>This sounds to me like making it more complicated. If you introduce
>random ordering of flow start times, you’ll need to make a choice for the
>interval over which randomness applies. On what basis? If it gets large,
>you can be back at flows gradually starting again.
>
>All in all, I’m not sure I agree if it makes sense to say that flows
>gradually starting “muddles” the behavior - you could also say that
>starting at the same time is a special case…


[XZ] Yes, starting at the same time is a special case. But that’s the case
that does not give any non-RTT-related bias for any flows. Since we care
about RTT-fairness in Test Case 5.5, I thought it’s important to design
the test case to avoid those biases.  However, we do not need to stick
with starting at same time.  We could also use random starting older to
avoid such bias.   



[XZ] Let’s go with the concrete example with 5 flows with RTT of 20ms,
50ms, 100ms, 200ms, and 300ms. If they start sequentially, 10 seconds
apart (as specified in the current draft), then the flow with RTT of 300ms
will get a higher rate than the first flow with 20ms even if the
underlying CC algorithm is completely RTT fair. If, instead, we run the
test several times with different starting *order* yet deterministic
starting time (e.g., flow w/ RTT=300ms starting first, flow w/ RTT=20ms
starting second) then that effect will not show up in the final averaged
rate per flow. 



>
>
>>>> 	€ Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue
>>>> sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes
>>>> (CoDel and PIE) are striving to contain queuing delay with a target of
>>>> 20ms, while allowing for a larger physical queue size to accommodate
>>>> instantaneous bursts, a value of 20m for a drop-tail queue depth seems
>>>> rather out of place.  Should we simply remove the value of 20ms for
>>>>now?
>>>> If we want testers to play with a lower queue depth, we may start with
>>>> 100ms of queuing, which corresponds to the bandwidth-delay product
>>>>along
>>>> that path (50ms propagation each direction).  On a related note: the
>>>> queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is
>>>> specified as 70ms. Just wondering how that number came about?
>>> 
>>> I have no strong opinion on this one; though, in a way I'd say it's a
>>> good thing that the 20ms case highlights what happens when the queue is
>>> below the target, we should at least know what controllers do under
>>>such
>>> circumstances?  But then this may just twist the result strangely if
>>>it's
>>> part of a test case, not a test case on its own, so I can understand
>>>that
>>> it seems out of place, as you say...
>> 
>> 
>> [XZ] Not sure if I understand what you mean by ³when the queue is below
>> the target²?
>
>I meant, what if e.g. you have a 10ms queue but congestion control
>mechanisms try to operate at 20ms?
>(20 / 20 is a special case at the borderline between this (“below the
>target”) and “normal”)
>

[XZ]  Thanks for clarifying this. Yes, will be interesting to find out. As
an extension, will also be interesting to see how CC schemes with a target
queue delay (e.g., 50ms) interact with an AQM that tries to keep the
queuing delay at 20ms.

>
>> How would source congestion control behave if we indeed have
>> a bottleneck queue depth of 20ms? I¹d say people are welcome to test out
>> that scenario. Just that the setup itself does not seem to be motivated
>>by
>> a practical deployment scenario.
>> 
>> [XZ] In any case I suppose we share the same viewpoint on this one.
>
>Yes
>
>Cheers,
>Michael
>