Re: [rmcat] Review of draft-ietf-rmcat-eval-test-02

[Zaheduzzaman Sarker <[e-mail]>]

Hi Safiqul,

Thanks for your review of the documents.

please see inline below.

BR

Zahed

On 2016-01-27 16:47, Safiqul Islam wrote:
> Hi all,
>
> Here are my comments for eval-test draft. The draft provides sufficient
> details of the basic test scenarios to evaluate the candidate algorithms.
>
> General comments:
>
>   * The abstract can clarify the scope of the test scenarios.

could you explain a but more what needs to be clarified?

>   * The document uses on the following terms interchangeably to
>     represent RMCAT flows. I would like to suggest using only one of
>     them in the document to make it consistent:
>
>      1.   Media flow
>      2.   RMCAT flow
>      3.   RMCAT media flow

Good point, I think we need a terminology section to clarify this.

>
>   * The document mentions “it is expected to log enough information,
>     e.g., Page 14, section 5.3,  “To evaluate the performance of the
>     candidate algorithms it is expected to log enough information “
>       o what is “it” ? It needs to be clarified.

I can also see the doc mentions "to visualize the metrics described in 
Section 4.1". So the document points to number of important metrics. As 
the document does not dictate in what environment the tests will be 
performed (simulator or testbed, real implementation or simulation), it 
is very hard to say how and what can be logged. I believe that is an 
implementation details that we don't need to describe here. But the text 
can be improved to clarify this.

>   * The document suggests to use bit rate range 150 kbps - 1.5 Mbps. I
>     think the upper limit should be increased to match with  real-life
>     applications.

I think we have discussed this in the past and settled with this bitrate 
range for the test cases. in section 4.3 the text explains this.

>
>
> Specific, miscellaneous comments/questions:
>
>   * Page 3, section 3, Expected behavior:,  describe -> describes
>   * I suggest using either behaviour or behavior to make it consistent.
>   * Page 3, last line, The following basic structure of test case -> of
>     a test case
>   * Page 4. line 2, I suggest using a different verb instead of “flow"
>     to avoid confusion with “flows"
>   * Page 4, line 11, The following sentence needs to be rephrased.
>
> It should be noted that depending on the test cases it is possible to
> have different path characteristics in of the either directions.
>
>   * Page 7, section 4.1,  metircs -> metrics
>   * Page 9, section 4.3,  The following sentence needs to be revised.
>
> "Also it is possible that the media traffic generator used in a
> particular  simulator or testbed if not capable of generating higher
> bitrate.”
>
>   * Page 10,  line 2, unless otherwise specified in the test case ->
>     unless otherwise specified in the test case(s)
>   * Page 10, an optimal startup  bitrate values -> an optimal startup
>       bitrate value
>   * Page 10, section 5.1, maximum Media Bit Rate is Greater than Link
>     Capacity -> maximum media bit rate is greater than link capacity
>   * Page 11, paragraph 2, the following sentence needs to be rephrased
>     to enhance clarity.
>
> The oscillations occur when the media flow(s) attempts to reach its
> maximum bit rate, overshoots the usage of the available bottleneck
> capacity, to rectify it reduces the bit rate and starts to ramp up again.

ok

>
>   * Page 12, the following sentence can be broken into smaller sentences
>     to enhance clarity.
>
> When using background non-adaptive UDP traffic to induce time-varying
> bottleneck for the RMCAT flow, the physical path capacity is 4Mbps and
> the UDP traffic source rate changes over  time as (4-x)Mbps, where x is
> the bottleneck capacity specified in Table 1

will try.

>
>   * Page 13, at a fine enough time granularity: -> at a fine enough time
>     granularity.
>   * Page 14, i.e, when the backward channel has no impairments -> i.e,
>     when the backward channel has no impairments.
>   * Page 14, section 5.3, at a fine-grained time intervals: -> at a
>     fine-grained time interval.
>   * Page 16, section 5.4, more than one RMCAT media flow shares the
>     -> more than one RMCAT media flow share the
>
>   * Page 17,  connected to respective R1, R2, R3. -> connected to their
>     respective destinations R1, R2, R3.
>
>   * Page 19,  It is expected that the competing flows will converge to
>     bit rates to ->  It is expected that the competing flows will
>     converge to optimum bit rates to
>   * Page 19, at a fine enough time granularity: -> at a fine enough time
>     granularity.
>   * Page 19,  One-Way propagation delay for each flow: 10ms, 25ms, 50ms,
>     100ms, 150ms. -> One-Way propagation delay for each flow: 10ms,
>     25ms, 50ms, 100ms, and 150ms respectively.
>   * Page 20, section 5.6, at least one long lived TCP flows -> at least
>     one long lived TCP flow
>   * Page 21, connected to corresponding media sink, R1 -> connected
>     to its corresponding media sink, R1
>   * Page 21,  The TCP traffic goes over the forward path from, S_tcp
>     with acknowledgement packets flowing along the backward path from,
>     R_tcp ->  The TCP traffic goes over the forward path from S_tcp, and
>     corresponding acknowledgement packets are transmitted over the
>     backward path from R_tcp.
>   * Page 22, one (1), long-lived TCP -> one (1) and long-lived TCP
>   * Page 22, Congestion control: Default TCP congestion control.
>     -> Congestion control: default TCP congestion control.
>   * Page 22, Test Specific Information: None -> Test Specific
>     Information: none
>   * Page 23, This scenario shows the performance of the multimedia
>     application -> This scenario shows the performance of a multimedia
>     application
>   * Page 23, Media Source:  - > Media source:
>   * Page 24,  The following sentence needs to be revised:
>       o Not all short TCPs start at the same time, 2 start in the ON
>         state while 8 start in an OFF stats.
>   * Page 25, section 5.8, At these stage one of the media flow is
>     -> At this stage, one of the media flows is
>   * Page 25, section 5.8, flows and as they -> flows as they
>   * Page 25. section 5.8,  The following sentence s

?? what is your comment here?
>
> When the paused media flow will resume it would no longer have the same
> bandwidth share on the link.  It has to make  it's way through the other
> existing flows in the link to achieve a fair share of the link capacity.

>
>   * Page 25, section 5.8, The general description of the testbed
>     parameters are -> The general description of the testbed parameters is
>   * Page 26, section 6.1, the candidate algorithms does not fail -> the
>     candidate algorithms do not fail
>   * Page 26, section 6.2, In this test case one RMCAT flow, S1->R2
>     traverse a path -> In this test case one RMCAT flow, S1->R2
>     traverses a path
>   * Page 27, connected to respective destinations R1, R2, and R3. ->
>     connected to their respective destinations R1, R2, and R3.
>
>   * I would like to suggest the following two additional test-cases
>     under section 6 (other potential test cases).

The test cases in the draft were designed for end-point adaptation 
algorithm without coupled congestion control in mind i.e. no need for 
shared bottleneck detection and flow priority. In the last IETF meeting 
we talked about the possibility of extending the test cases to include 
coupled congestion control algorithm.

Now if we extend the test cases then I don't think only adding the two 
proposed test cases will be sufficient. We will need to identify all the 
test cases those are suitable for running with coupled congestion 
control. Like 5.2 can be run with priority per flow. By doing this we 
can mention (in each test case or in one place in the document) that one 
can run tests with flow priority and perhaps give some means to evaluate 
the available capacity distribution with some equation like the one 
mentioned here in the proposed test case 6.3. However, then this 
document also needs to include description for shared bottleneck 
detection. In the simplest form, we can perhaps assume that the 
identified test cases have single bottleneck point.

I think we need to discuss what is best possible way to extend the test 
cases if we agreed to do that.

But before that I would like to know if the WG is OK with extending the 
test cases for coupled congestion control.

And will it be a compulsory (candidate algorithms MUST present results) 
or optional (candidate algorithms MAY present results) part of the test 
case.

>
> 6.3 Prioritization test
>
> In this test case, three RMCAT media flows started  out with a priority
> of 1 each. At around 50 s, the priorities of flow 2 and flow 3 were
> decreased to 1/3 and 2/3.  A flow i  is assigned the available capacity
> based on its priority p_{i} as follows:
>
> b_{i} = (p_{i} / Sum(p_{k}) ) * BW   ;  k = {1.. 3}
>
> This test case ensures that flows are getting their assigned portions of
> the available capacity based on their assigned priorities. It can be
> useful for [I-D.ietf-rmcat-coupled-cc]. It addresses the requirement 3A
> in  [I-D.ietf-rmcat-cc-requirements]
>
> Testbed topology: Same as test case defined in section 5.4
>
> Testbed Attribute:
>
> The general description of the testbed parameters is same as Section 5.4
> with changes in the test specific setup as  below-
>
>    o  Other test specific setup:
>
>        *  RMCAT media flow timeline:
>
>           +  Flow ID: One (1)
>
>           +  Start time: 0s
>
>           +  Flow duration: 119s
>
>           +  Priority: no change
>
>       *  RMCAT media flow timeline:
>
>           +  Flow ID: Two (2)
>
>           +  Start time: 0s
>
>           +  Flow duration: 119s
>
>           +  Priority:  change to 1/3 after 50s
>
>        *  RMCAT media flow timeline:
>
>           +  Flow ID: Three (3)
>
>           +  Start time: 0s
>
>           +  Flow duration: 119s
>
>           +  Priority: change to 2/3 after  50s
>
>
> 6.4 RMCAT flows competing with one or more long TCP flows
>
> This test case is an extension of the test case defined in section 5.6.
> It is important for [I-D.ietf-rmcat-coupled-cc], where it requires more
> than one RMCAT flows in order to couple them.
>
> The general description of the testbed parameters is same as section 5.6.
>
>
> Regards,
> Safiqul
>
>
>
>
>
>

-- 
Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER


Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone [phone]
Fax [phone]
SMS/MMS [phone]
[e-mail]
www.ericsson.com

==================================================