Re: [rmcat] Review of draft-ietf-rmcat-eval-test-02

[Safiqul Islam <[e-mail]>]

Hi Zahed,

Comments inline.


On 28. jan. 2016, at 14.29, Zaheduzzaman Sarker <[e-mail]<mailto:[e-mail]>> wrote:

Hi Safiqul,

Thanks for your review of the documents.

please see inline below.

BR

Zahed

On 2016-01-27 16:47, Safiqul Islam wrote:
Hi all,

Here are my comments for eval-test draft. The draft provides sufficient
details of the basic test scenarios to evaluate the candidate algorithms.

General comments:

 * The abstract can clarify the scope of the test scenarios.

could you explain a but more what needs to be clarified?

if you mention that the test cases cover the basic usage scenarios and can easily be extended, it will add value for the reader.



 * The document uses on the following terms interchangeably to
   represent RMCAT flows. I would like to suggest using only one of
   them in the document to make it consistent:

    1.   Media flow
    2.   RMCAT flow
    3.   RMCAT media flow

Good point, I think we need a terminology section to clarify this.


 * The document mentions “it is expected to log enough information,
   e.g., Page 14, section 5.3,  “To evaluate the performance of the
   candidate algorithms it is expected to log enough information “
     o what is “it” ? It needs to be clarified.

I can also see the doc mentions "to visualize the metrics described in Section 4.1". So the document points to number of important metrics. As the document does not dictate in what environment the tests will be performed (simulator or testbed, real implementation or simulation), it is very hard to say how and what can be logged. I believe that is an implementation details that we don't need to describe here. But the text can be improved to clarify this.

 * The document suggests to use bit rate range 150 kbps - 1.5 Mbps. I
   think the upper limit should be increased to match with  real-life
   applications.

I think we have discussed this in the past and settled with this bitrate range for the test cases. in section 4.3 the text explains this.

OK.



Specific, miscellaneous comments/questions:

 * Page 3, section 3, Expected behavior:,  describe -> describes
 * I suggest using either behaviour or behavior to make it consistent.
 * Page 3, last line, The following basic structure of test case -> of
   a test case
 * Page 4. line 2, I suggest using a different verb instead of “flow"
   to avoid confusion with “flows"
 * Page 4, line 11, The following sentence needs to be rephrased.

It should be noted that depending on the test cases it is possible to
have different path characteristics in of the either directions.

 * Page 7, section 4.1,  metircs -> metrics
 * Page 9, section 4.3,  The following sentence needs to be revised.

"Also it is possible that the media traffic generator used in a
particular  simulator or testbed if not capable of generating higher
bitrate.”

 * Page 10,  line 2, unless otherwise specified in the test case ->
   unless otherwise specified in the test case(s)
 * Page 10, an optimal startup  bitrate values -> an optimal startup
     bitrate value
 * Page 10, section 5.1, maximum Media Bit Rate is Greater than Link
   Capacity -> maximum media bit rate is greater than link capacity
 * Page 11, paragraph 2, the following sentence needs to be rephrased
   to enhance clarity.

The oscillations occur when the media flow(s) attempts to reach its
maximum bit rate, overshoots the usage of the available bottleneck
capacity, to rectify it reduces the bit rate and starts to ramp up again.

ok


 * Page 12, the following sentence can be broken into smaller sentences
   to enhance clarity.

When using background non-adaptive UDP traffic to induce time-varying
bottleneck for the RMCAT flow, the physical path capacity is 4Mbps and
the UDP traffic source rate changes over  time as (4-x)Mbps, where x is
the bottleneck capacity specified in Table 1

will try.


 * Page 13, at a fine enough time granularity: -> at a fine enough time
   granularity.
 * Page 14, i.e, when the backward channel has no impairments -> i.e,
   when the backward channel has no impairments.
 * Page 14, section 5.3, at a fine-grained time intervals: -> at a
   fine-grained time interval.
 * Page 16, section 5.4, more than one RMCAT media flow shares the
   -> more than one RMCAT media flow share the

 * Page 17,  connected to respective R1, R2, R3. -> connected to their
   respective destinations R1, R2, R3.

 * Page 19,  It is expected that the competing flows will converge to
   bit rates to ->  It is expected that the competing flows will
   converge to optimum bit rates to
 * Page 19, at a fine enough time granularity: -> at a fine enough time
   granularity.
 * Page 19,  One-Way propagation delay for each flow: 10ms, 25ms, 50ms,
   100ms, 150ms. -> One-Way propagation delay for each flow: 10ms,
   25ms, 50ms, 100ms, and 150ms respectively.
 * Page 20, section 5.6, at least one long lived TCP flows -> at least
   one long lived TCP flow
 * Page 21, connected to corresponding media sink, R1 -> connected
   to its corresponding media sink, R1
 * Page 21,  The TCP traffic goes over the forward path from, S_tcp
   with acknowledgement packets flowing along the backward path from,
   R_tcp ->  The TCP traffic goes over the forward path from S_tcp, and
   corresponding acknowledgement packets are transmitted over the
   backward path from R_tcp.
 * Page 22, one (1), long-lived TCP -> one (1) and long-lived TCP
 * Page 22, Congestion control: Default TCP congestion control.
   -> Congestion control: default TCP congestion control.
 * Page 22, Test Specific Information: None -> Test Specific
   Information: none
 * Page 23, This scenario shows the performance of the multimedia
   application -> This scenario shows the performance of a multimedia
   application
 * Page 23, Media Source:  - > Media source:
 * Page 24,  The following sentence needs to be revised:
     o Not all short TCPs start at the same time, 2 start in the ON
       state while 8 start in an OFF stats.
 * Page 25, section 5.8, At these stage one of the media flow is
   -> At this stage, one of the media flows is
 * Page 25, section 5.8, flows and as they -> flows as they
 * Page 25. section 5.8,  The following sentence s

?? what is your comment here?

So sorry, I forgot to complete the sentence. The following sentences should be revised to enhance clarity.


When the paused media flow will resume it would no longer have the same
bandwidth share on the link.  It has to make  it's way through the other
existing flows in the link to achieve a fair share of the link capacity.


 * Page 25, section 5.8, The general description of the testbed
   parameters are -> The general description of the testbed parameters is
 * Page 26, section 6.1, the candidate algorithms does not fail -> the
   candidate algorithms do not fail
 * Page 26, section 6.2, In this test case one RMCAT flow, S1->R2
   traverse a path -> In this test case one RMCAT flow, S1->R2
   traverses a path
 * Page 27, connected to respective destinations R1, R2, and R3. ->
   connected to their respective destinations R1, R2, and R3.

 * I would like to suggest the following two additional test-cases
   under section 6 (other potential test cases).

The test cases in the draft were designed for end-point adaptation algorithm without coupled congestion control in mind i.e. no need for shared bottleneck detection and flow priority. In the last IETF meeting we talked about the possibility of extending the test cases to include coupled congestion control algorithm.

Now if we extend the test cases then I don't think only adding the two proposed test cases will be sufficient. We will need to identify all the test cases those are suitable for running with coupled congestion control. Like 5.2 can be run with priority per flow. By doing this we can mention (in each test case or in one place in the document) that one can run tests with flow priority and perhaps give some means to evaluate the available capacity distribution with some equation like the one mentioned here in the proposed test case 6.3. However, then this document also needs to include description for shared bottleneck detection.

I think showing prioritization results with test-case 5.4 should be sufficient.



In the simplest form, we can perhaps assume that the identified test cases have single bottleneck point.


+1

I think we need to discuss what is best possible way to extend the test cases if we agreed to do that.

But before that I would like to know if the WG is OK with extending the test cases for coupled congestion control.

And will it be a compulsory (candidate algorithms MUST present results) or optional (candidate algorithms MAY present results) part of the test case.

The two test cases are written explicitly to highlight the features of coupled congestion control where we needed to play around with more than one flow and priorities.
In my opinion, the rest of the test cases can remain as they are, and have no direct effect from coupled congestion control. That’s why I recommended to
add them in section 6 (other potential test cases).



6.3 Prioritization test

In this test case, three RMCAT media flows started  out with a priority
of 1 each. At around 50 s, the priorities of flow 2 and flow 3 were
decreased to 1/3 and 2/3.  A flow i  is assigned the available capacity
based on its priority p_{i} as follows:

b_{i} = (p_{i} / Sum(p_{k}) ) * BW   ;  k = {1.. 3}

This test case ensures that flows are getting their assigned portions of
the available capacity based on their assigned priorities. It can be
useful for [I-D.ietf-rmcat-coupled-cc]. It addresses the requirement 3A
in  [I-D.ietf-rmcat-cc-requirements]

Testbed topology: Same as test case defined in section 5.4

Testbed Attribute:

The general description of the testbed parameters is same as Section 5.4
with changes in the test specific setup as  below-

  o  Other test specific setup:

      *  RMCAT media flow timeline:

         +  Flow ID: One (1)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority: no change

     *  RMCAT media flow timeline:

         +  Flow ID: Two (2)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority:  change to 1/3 after 50s

      *  RMCAT media flow timeline:

         +  Flow ID: Three (3)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority: change to 2/3 after  50s


6.4 RMCAT flows competing with one or more long TCP flows

This test case is an extension of the test case defined in section 5.6.
It is important for [I-D.ietf-rmcat-coupled-cc], where it requires more
than one RMCAT flows in order to couple them.

The general description of the testbed parameters is same as section 5.6.


Regards,
Safiqul






Cheers,
Safiqul