Re: [rmcat] WG last call draft-ietf-rmcat-wireless-tests-06

["Xiaoqing Zhu (xiaoqzhu)" <[e-mail]>]

Thanks a lot, Sergio, for reviewing our updated draft, and also for providing further suggestions.

WG Chairs – would you recommend us to update the draft again to incorporate these editorial changes, hence updating the draft from -07 to -08?  Or shall we hold on to these (fairly local and minor) changes and combine them in the next round of revision?  Please advise.

Thanks,
Xiaoqing

From: Sergio Mena <[e-mail]>
Date: Tuesday, July 2, 2019 at 8:17 AM
To: Xiaoqing Zhu <[e-mail]>
Cc: "[e-mail]" <[e-mail]>, Zaheduzzaman Sarker <[e-mail]>, Ingemar Johansson S <[e-mail]>, "Dan Tan (dtan2)" <[e-mail]>, "Michael Ramalho (mramalho)" <[e-mail]>, "Jeromy Fu (jianfu)" <[e-mail]>
Subject: Re: [rmcat] WG last call draft-ietf-rmcat-wireless-tests-06


Hi Xiaoqing, draft authors,

Thanks for addressing my comments. I carefully went through your answers and the updated draft. In general, I agree with the way you addressed the comments.

I only have three extra comments to make. Two of them are editorial fixes I caught while going through the new version, and the third is a reaction to one of your answers (please see inline [SM]).

Editorial:

* (über-minor) Section 4. Fifth bullet. "[Heusse2003]since" --> "[Heusse2003] since" (missing space)

* Section 4.2.4. Last bullet. "throughtput" --> "throughput"

Independently of those extra comments, I believe that, overall, my review has been properly addressed and is thus complete.

I hope my review helped improve the draft's quality.

Thanks,

Sergio


On 01/07/2019 20:26, Xiaoqing Zhu (xiaoqzhu) wrote:


Hi Sergio,



Thanks again for your valuable review for the draft.  We finally got a chance to update the draft to version -07 so as to incorporate your comments. Apologies for the long wait!



https://tools.ietf.org/html/draft-ietf-rmcat-wireless-tests-07



Please find below inline detailed responses (tagged [Authors]). And, of course, any further comments or discussions are very welcome.



Cheers,

Xiaoqing (on behalf of all authors)



On 2/6/19, 9:06 AM, "rmcat on behalf of Sergio Mena" <[e-mail] on behalf of [e-mail]><mailto:[e-mail]@cisco.com> wrote:



    RMCAT working group,



    This is my review of draft-ietf-rmcat-wireless-tests-06. I realize my

    review comes a few days after the deadline. Sorry about that. Hopefully

    it's not too late.



    I have structured my feedback in three parts: minor/editorial comments,

    "less minor" comments, and general comments.



    * General comments.

       - Although I had some (fairly minor) suggestions on the particular

    setup of test cases, I understand it is too late for that, as the draft

    is quite mature and several teams have already implemented the TCs as

    they are described, so this review doesn't attempt to re-discuss the

    values/topologies chosen for the test cases.

       - Wifi6. The document does not make any reference to Wifi6. I am

    aware of the fact the the standard is not yet finalized, but I think the

    announced innovations (in particular OFDMA, and the increased scheduling

    flexibility it mail entail) would justify at least a general mention. My

    suggestion is that page 11 could be a good place to introduce such a

    reference. Section 4.3 is another good spot.





 [Authors] Thanks for the great suggestion.  Sec. 4 is now revised as follows. Note that we've also

updated the statistics for 11ac and 11n to catch up on their evolution over time.



  Unless otherwise mentioned, test cases in this section are described

         using the underlying PHY- and MAC-layer parameters based on the IEEE

         802.11n Standard.  Statistics collected from enterprise Wi-Fi

         networks show that the two dominant physical modes are 802.11n and

         802.11ac, accounting for 41% and 58% of connected devices.  As Wi-Fi

         standards evolve over time, for instance, with the introduction of

         the emerging Wi-Fi 6 (802.11ax) products, the PHY- and MAC-layer test

         case specifications need to be updated accordingly to reflect such changes.



  ...



  All test cases described below can be carried out using simulations,

         e.g. based on [ns-2] or [ns-3].  When feasible, it is also encouraged

         to perform testbed-based evaluations using Wi-Fi access points and

         endpoints running up-to-date IEEE 802.11 protocols, such as 802.11ac

         and the emerging Wi-Fi 6, to verify the viability of the candidate

         schemes.



    * "less minor"

       - Page 6. Section 3.1.1. Please do not define the bandwidth as

    "infinite". That is problematic in a simulation. Rather, choose a big

    enough value (e.g., 10 Gbps) that is big enough so that it will not

    produce any congestion.



[Authors]  Thanks for raising this point.  The text in that section is now revised as:



  ... The fixed user is connected to the Internet via wired connection

  with sufficiently high bandwidth, for instance, 10 Gbps, so that the

  system is resource-limited on the wireless interface.



       - Page 10. Last bullet. I would use the expression "non-congestion

    related losses" in this bullet, to be consistent with the cellular section.





[Authors]  We've discussed offline regarding this topic.  In order to describe

the nature of packet losses over Wi-Fi more accurately, the wording in the 3rd

bullet point of Sec. 4 has been revised to:



  * Packet transmissions over Wi-Fi are susceptible to contentions and

         collisions over the air.  Consequently, traffic load beyond a

         certain utilization level over a Wi-Fi network can introduce

  frequent collisions over the air and significant network overhead,

         as well as packet drops due to buffer overflow at the transmitters.

  This, in turn, leads to excessive delay, retransmissions, packet losses

  and lower effective bandwidth for applications.  Note, however, that

  the consequent delay and loss patterns caused by collisions are

  qualitatively different from those induced by congestion over a wired

  connection.



       - Page 11. Last paragraph. I wonder why the cellular section

    recommends or suggests that the tests should be carried out only in a

    simulator, whereas the WiFi section recommends to also run some

    "evaluation" in a real testbed.





[Authors] Thanks for raising this important issue. Even though it is possible to carry out tests

over LTE (and 5G) networks, and actually it is already done so today, these tests cannot in the

general case be carried out in a deterministic fashion or to ensure repeatability, as these network

are in the control of cellular operators and there can various amounts of competing traffic in the

same cell(s). In practice, it is only in underground mines that one can carry out near deterministic

testing. Even there, it is not guaranteed either as workers in the mines carry with them their

iPhones and Androids.  Also, the underground mining setting may not reflect typical usage

patterns of a urban setting.



[Authors] The main reason for the lack of suggestion for testbed-based evaluations for cellular

in the draft is out of the above concerns of practical feasibility,  whereas Wi-Fi equipment is

relatively lower cost to acquire and setup.  Based on the reviewer comments, we've revised the

discussion in this paragraph (Sec. 4) to tune down the level of recommendation for testbed-based

Wi-Fi evaluations to "encourage":



  All test cases described below can be carried out using simulations,

         e.g. based on [ns-2] or [ns-3].  When feasible, it is also encouraged

         to perform testbed-based evaluations using Wi-Fi access points and

         endpoints running up-to-date IEEE 802.11 protocols, such as 802.11ac

         and the emerging Wi-Fi 6, to verify the viability of the candidate schemes.
[SM] Thanks for the explanation. I find it valuable. Please consider explaining this in the draft, e.g., by squeezing a few sentences somewhere in the cellular section (suggestion: at the end of page 5).




       - Page 16. "Congestion control" bullet includes the acronym [TBD].

    Does it mean "to be decided"? If it does, what is the decision? BTW, why

    aren't TCP flows used in the cellular TCs (or did I miss them?)







[Authors] Good catch. This was indeed a "to-be-decided" reminder to the editors.

We've updated the reference for the default congestion control scheme of TCP to

[RFC5681] so as to align with the eval-test draft

(https://tools.ietf.org/html/draft-ietf-rmcat-eval-test-10, see Sec. 5.6 and 5.7).



The draft now specifies background FTP traffic over TCP (using default congestion

control per RFC5681) in both Sections 3.1 and 3.2 for cellular test cases.



    * minor/editorial

       - Abstract. "These applications are typically required to implement

    congestion control". Grammar is correct, but somewhat ambiguous on what

    requires what. Suggestion: "A congestion control algorithm is typically

    required by these applications"

       - Page 4. 1st paragraph. "Besides, there are certain characteristics

    which make the cellular network different and challenging than other

    types of access network such as Wi-Fi and wired network”. Please fix

    grammar near the word "than".

       - Page 5. "The key factors to define test cases for cellular network

    are" should read either "for a cellular network", or "for cellular

    networks". There are other occurrences of this further down in the text

       - Page 6. In my opinion, "there should be enough amount of time"

    could be shortened to "there should be enough time"

       - Page 8. "intercity" --> "intensity" (?)

       - Page 11. First paragraph "singal" --> "signal"

       - Page 11. "Throughout this draft". I would rather say "Throughout

    the 'WiFi networks' section" or "Throughout this section" something

    similar, to avoid confusion with the cellular section

       - Page 12. 1st paragraph. "set up" --> "setup"

       - All over. Search and replace: "traffics" --> "traffic"

       - All over. Search and replace "followings" --> "following"



[Authors]  Thanks for identifying all these nits via your "minor/editorial" comments. The

draft has been updated accordingly.





    I hope this will help improve the quality of the text.



    Sergio





    On 15/01/19 00:24, Colin Perkins wrote:

    > This is to announce a working group last call on the “Evaluation Test Cases for Interactive Real-Time Media over Wireless Networks” (draft-ietf-rmcat-wireless-tests-06).

    >

    > Please send any final comments to the working group mailing list and the authors by 1 February 2019. If no substantive comments are received by that time, we intend to submit this draft to the IESG for publication as an Informational RFC.

    >

    > Colin

    > (as WG co-chair)

    >

    >

    >

    >

    >