Re: [rmcat] Updates for Test Case 5.5 and 5.6 in draft-ietf-rmcat-eval-test

[Michael Welzl <[e-mail]>]

> On 03 Sep 2015, at 22:22, Xiaoqing Zhu (xiaoqzhu) <[e-mail]> wrote:
> 
> Hi all, 
> 
> We are about to update the test case draft to keep it alive by September 11, 2015. And, as co-author, I would like to poll the mailing list regarding two proposed changes for Test Case 5.5 and 5.6.  
> 	• Test Case 5.5 (RTT Fairness):  currently all flows start gradually, and that may lead to late-coming flows settling at higher rates if they rely on queuing delay measurements.  Since the intention of this test case is to measure fairness w.r.t RTT, would it make more sense to start all flows at the same time, or roughly the same time (e.g., random start within 5 seconds) to avoid flow synchronization effects?  Notice that if any candidate scheme has an issue with late-comer advantage, that should have already been exposed by Test Case 5.4. 

I'd say no to this one: I don't understand how starting at the same time or roughly at the same time would avoid flow synchronization effects?  RTT Fairness should clearly work if flows join later or all at roughly the same time. Also note that randomness only makes sense to introduce if there is a minimum number of repetitions and we consider the distribution of output data, i.e. it's not particularly useful when showing plots over time as is the common thing here.


> 	• Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes (CoDel and PIE) are striving to contain queuing delay with a target of 20ms, while allowing for a larger physical queue size to accommodate instantaneous bursts, a value of 20m for a drop-tail queue depth seems rather out of place.  Should we simply remove the value of 20ms for now?   If we want testers to play with a lower queue depth, we may start with 100ms of queuing, which corresponds to the bandwidth-delay product along that path (50ms propagation each direction).  On a related note: the queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is specified as 70ms. Just wondering how that number came about? 

I have no strong opinion on this one; though, in a way I'd say it's a good thing that the 20ms case highlights what happens when the queue is below the target, we should at least know what controllers do under such circumstances?  But then this may just twist the result strangely if it's part of a test case, not a test case on its own, so I can understand that it seems out of place, as you say...


Cheers,
Michael