[rmcat] Review on draft-ietf-rmcat-eval-test-02

[Stefan Holmer <[e-mail]>]

Hi,

Sorry for being late with this review, here are my comments:

*General comments:*

   - We consider audio and video content, but should we also consider slide
   shows and/or screen content? In my experience those often sources are
   widely different in their rate characteristics, and may be very bursty. I
   think it could be interesting to see how different candidates handle the
   case where they are source limited most of the time (static image, encoder
   produces < 50 kbps), and every now and then there's a slide change.
   - Each video stream is often accompanied by a constant bitrate audio
   stream in these test cases, but it's not clear to me why? I think we need
   to clarify how the audio stream is going to be evaluated. Are we going to
   look at end-to-end latency and jitter?
   - It's suggested that video resolution should be specified in the test
   cases, but we don't seem to suggest that video quality should be evaluated
   (PSNR/SSIM). How are we then expecting that video resolution will matter in
   these tests? Do we allow the encoder to internally scale down the
   resolution depending on the available bandwidth? In that case, will be
   really be evaluating CC candidates or are we starting to lean towards
   comparing full RTC systems?


*Raw comments:*

     4.1.  Evaluation metircs  . . . . . . . . . . . . . . . . . . .   7
Comments:

   - metrics


*Section 3*

It should be noted that
      depending on the test cases it is possible to have different path
      characteristics in of the either directions.

Comments:

   - "in either of the directions"
   - There is a bullet which looks misplaced after this.


In a  testbed environment laboratory there may exist a significant
      amount of traffic on portions of the network path between the
      endpoints that is not desired for the purposes of these RMCAT
      tests.

Comments:

   - I think we should define what a testbed environment laboratory is
   somewhere. Is it a LAN with some network emulator, or is it a fully
   simulated network?


  +  Bottleneck queue size: defines size of queue in terms of
            queuing time when the queue is full (in milliseconds).

Comments:

   - "the size of the queue"


      *  Application-related: defines the traffic source behaviour for
         implementing the test case

Comments:

   - In this section we mention Video/Voice as media types. Should we also
   consider screen content/slide shows? See general comments.
   - Under adaptability we mention resolution as being an option of
   adaptation. It's not clear to me how we should evaluate something like that
   without comparing the visual quality in some way. See general comments.


*Section 4.3*

Also it is possible
               that the media traffic generator used in a particular
               simulator or testbed if not capable of generating higher
               bitrate.  Hence we have selected a suitable bitrate range

Comments:

   - Spelling errors:
      - "testbed is not"
      - "bit rate"

*Section 5.1*

maximum Media Bit Rate is Greater than Link Capacity.  In this
      case, the application will attempt to ramp up to its maximum bit
      rate, since the link capacity is limited to a value lower, the
      congestion control scheme is expected to stabilize the sending bit
      rate close to the available bottleneck capacity.  This situation
      can occur when the endpoints are connected via thin long networks
      even though the advertised capacity of the access network may be
      higher.

Comments:

   - I don't understand the last sentence here. Isn't it simpler to refer
   to e.g. cable/adsl links where the uplink may be in the order of 256 - 1024
   kbps.


   It should be noted that the exact variation in available capacity due
   to any of the above depends on the under-lying technologies.  Hence,
   we describe a set of known factors, which may be extended to devise a
   more specific test case targeting certain behaviour in a certain
   network environment.

Expected behavior: the candidate algorithm is expected to detect the
   path capacity constraint, converges to bottleneck link's capacity and

Comments:

   - "underlying"
   - "targeting* a* certain"
   - "converges to *the* bottleneck link's capacity"


   o  Path characteristics: as described in Section 4.2

Comments:

   - I don't think there is a point in mentioning that we are using the
   defaults.

      *  This test uses the following one way propagation delays of 50
         ms and 100 ms.

Comments:

   - "This test uses one way propagation delays of 50 ms and 100 ms"
   - Is this referring to two instances of the test, or to forward and
   backward delays?


*Section 5.2*

   Expected behavior: the candidate algorithms is expected to detect the
   variation in available capacity and adapt the media stream(s)
   accordingly.  The flows stabilize around their maximum bitrate as the
   as the maximum link capacity is large enough to accommodate the
   flows.  When the available capacity drops, the flow(s) adapts by
   decreasing its sending bit rate, and when congestion disappears, the
   flow(s) are again expected to ramp up.

Comments:

   - spelling error "maximum bitrate as the maximum link capacity"
   - Rewrite last sentence as plural.


*Section 5.3*

   It is expected that the candidate algorithms is able to cope with the
   lack of feedback information and adapt to minimize the performance
   degradation of media flows in the forward channel.

   It should be noted that for this test case: logs are compared with
   the reference case, i.e, when the backward channel has no impairments

Comments:

   - "algorithms *are* able"
   - End last sentence with ".
   - I think the test duration should be longer since this is a fairly
   complicated scenario. What about 300 s?


*Section 5.4*

   In this test case, more than one RMCAT media flow shares the
   bottleneck link and each of them uses the same congestion control
   algorithm.  This is a typical scenario where a real-time interactive
   application sends more than one media flows to the same destination
   and these flows are multiplexed over the same port.

Comments:

   -
   - I don't think multiplexing is a good example here. There is no reason
   why they would actually fight for bandwidth between each other if they are
   multiplexed over the same port, as they may as well run a single CC and
   allocate the available bandwidth between them. Better to take an example of
   two or more different endpoints in the same office calling two or more
   endpoints in another office, or similar.
   - misspelled: "one media flows" should be "one media flow"


   Testbed topology: Three media sources S1, S2, S3 are connected to
   respective R1, R2, R3.

Comments:

   - Should probably be "are connected to R1, R2 and R3 respectively".



             +---------+------------+------------+----------+
             | Flow IF | Media type | Start time | End time |
             +---------+------------+------------+----------+

Comments:

   - Flow ID


*Section 5.1*

      *  Path capacity ratio: 1.0
Comments:

   - Not necessary to mention as I think it is the default? Also mentioned
   in other places.

         +  Congestion control: Default TCP congestion control.
Comments:

   - Should we suggest what the default TCP CC could be? New reno, Cubic?

*Section 5.8*

   In this test case, more than one real-time interactive media flows
   share the link bandwidth and all flows reach to a steady state by
   utilizing the link capacity in an optimum way.  At these stage one of
   the media flow is paused for a moment.  This event will result in
   more available bandwidth for the rest of the flows and as they are on
   a shared link.  When the paused media flow will resume it would no
   longer have the same bandwidth share on the link.  It has to make
   it's way through the other existing flows in the link to achieve a
   fair share of the link capacity.  This test case is important
   specially for real-time interactive media which consists of more than
   one media flows and can pause/resume media flow at any point of time
   during the session.  This test case directly addresses the
   requirement number 5 in [I-D.ietf-rmcat-cc-requirements].  One can
   think it as a variation of test case defined in Section 5.4.
   However, it is different as the candidate algorithms can use
   different strategies to increase its efficiency, for example the
   fairness, convergence time, reduce oscillation etc, by capitalizing
   the fact that they have previous information of the link.

Comments:

   - Several spelling errors:
      - "At *this* stage one of  the media *flows* is paused for a moment."
      -
      - "This event will result in more available bandwidth for the rest of
      the flows *(and)* as they are on a shared link.  When the
      paused media flow *resumes* it *will* no longer have the same
      bandwidth share on the link."
      -
      - " ... one media flows and can pause/resume media *flows* at any
      point of time during the session."
      -
      - "However, it is different as the candidate algorithms can use
      different strategies to increase *their* efficiency, for example
      *(the)* fairness"


*Section 6*
 It has been noticed that there are other interesting test cases
   besides the basis test cases listed above.

Comments:

   - "base"

*Section 6.2*
   Testbed attributes:

   o  Test duration: 120s

   o  Path characteristics:

      *  Reference bottleneck capacity between A and B = 2Mbps.
...

      *  One-Way propagation delay:

         1.  Between S1 and R1: 100ms

         2.  Between S2 and R2: 40ms

         3.  Between S3 and R3: 40ms

Comments:

   - Duration should probably be at least 300s given the complexity of this
   setup.
   - Does the reference bottleneck capacity apply to all links or only A
   and B? If only A and B, what's the reference for the other links?
   - In my opinion this test case is complex enough without different
   one-way propagation delays on the links.

*Section 7*
   Additional wireless network (both cellular network and WiFi network)
   specific test cases are define in [I-D.ietf-rmcat-wireless-tests]

Comments:

   - "defined in"