Re: [rmcat] WG last call draft-ietf-rmcat-wireless-tests-06

[Colin Perkins <[e-mail]>]

Hi,

If it’s easy to do, I’d recommend you make the editorial fixes now.

Colin



> On 2 Jul 2019, at 15:04, Xiaoqing Zhu (xiaoqzhu) <[e-mail]> wrote:
> 
> Thanks a lot, Sergio, for reviewing our updated draft, and also for providing further suggestions.
>  
> WG Chairs – would you recommend us to update the draft again to incorporate these editorial changes, hence updating the draft from -07 to -08?  Or shall we hold on to these (fairly local and minor) changes and combine them in the next round of revision?  Please advise. 
>  
> Thanks,
> Xiaoqing
>  
> From: Sergio Mena <[e-mail] <mailto:[e-mail]>>
> Date: Tuesday, July 2, 2019 at 8:17 AM
> To: Xiaoqing Zhu <[e-mail] <mailto:[e-mail]>>
> Cc: "[e-mail] <mailto:[e-mail]>" <[e-mail] <mailto:[e-mail]>>, Zaheduzzaman Sarker <[e-mail] <mailto:[e-mail]>>, Ingemar Johansson S <[e-mail] <mailto:[e-mail]>>, "Dan Tan (dtan2)" <[e-mail] <mailto:[e-mail]>>, "Michael Ramalho (mramalho)" <[e-mail] <mailto:[e-mail]>>, "Jeromy Fu (jianfu)" <[e-mail] <mailto:[e-mail]>>
> Subject: Re: [rmcat] WG last call draft-ietf-rmcat-wireless-tests-06
>  
> Hi Xiaoqing, draft authors,
> Thanks for addressing my comments. I carefully went through your answers and the updated draft. In general, I agree with the way you addressed the comments.
> I only have three extra comments to make. Two of them are editorial fixes I caught while going through the new version, and the third is a reaction to one of your answers (please see inline [SM]).
> Editorial:
> * (über-minor) Section 4. Fifth bullet. "[Heusse2003]since" --> "[Heusse2003] since" (missing space)
> * Section 4.2.4. Last bullet. "throughtput" --> "throughput"
> Independently of those extra comments, I believe that, overall, my review has been properly addressed and is thus complete.
> I hope my review helped improve the draft's quality.
> Thanks,
> Sergio
>  
> On 01/07/2019 20:26, Xiaoqing Zhu (xiaoqzhu) wrote:
> 
>> Hi Sergio, 
>>  
>> Thanks again for your valuable review for the draft.  We finally got a chance to update the draft to version -07 so as to incorporate your comments. Apologies for the long wait!
>>  
>> https://tools.ietf.org/html/draft-ietf-rmcat-wireless-tests-07 <https://tools.ietf.org/html/draft-ietf-rmcat-wireless-tests-07>
>>  
>> Please find below inline detailed responses (tagged [Authors]). And, of course, any further comments or discussions are very welcome. 
>>  
>> Cheers,
>> Xiaoqing (on behalf of all authors) 
>>  
>> On 2/6/19, 9:06 AM, "rmcat on behalf of Sergio Mena" <[e-mail] on behalf of [e-mail]> <mailto:[e-mail]@cisco.com> wrote:
>>  
>>     RMCAT working group,
>>     
>>     This is my review of draft-ietf-rmcat-wireless-tests-06. I realize my 
>>     review comes a few days after the deadline. Sorry about that. Hopefully 
>>     it's not too late.
>>     
>>     I have structured my feedback in three parts: minor/editorial comments, 
>>     "less minor" comments, and general comments.
>>     
>>     * General comments.
>>        - Although I had some (fairly minor) suggestions on the particular 
>>     setup of test cases, I understand it is too late for that, as the draft 
>>     is quite mature and several teams have already implemented the TCs as 
>>     they are described, so this review doesn't attempt to re-discuss the 
>>     values/topologies chosen for the test cases.
>>        - Wifi6. The document does not make any reference to Wifi6. I am 
>>     aware of the fact the the standard is not yet finalized, but I think the 
>>     announced innovations (in particular OFDMA, and the increased scheduling 
>>     flexibility it mail entail) would justify at least a general mention. My 
>>     suggestion is that page 11 could be a good place to introduce such a 
>>     reference. Section 4.3 is another good spot.
>>     
>>  
>>  [Authors] Thanks for the great suggestion.  Sec. 4 is now revised as follows. Note that we've also
>> updated the statistics for 11ac and 11n to catch up on their evolution over time.
>>  
>>   Unless otherwise mentioned, test cases in this section are described
>>          using the underlying PHY- and MAC-layer parameters based on the IEEE
>>          802.11n Standard.  Statistics collected from enterprise Wi-Fi
>>          networks show that the two dominant physical modes are 802.11n and
>>          802.11ac, accounting for 41% and 58% of connected devices.  As Wi-Fi
>>          standards evolve over time, for instance, with the introduction of
>>          the emerging Wi-Fi 6 (802.11ax) products, the PHY- and MAC-layer test
>>          case specifications need to be updated accordingly to reflect such changes.
>>   
>>   ...
>>   
>>   All test cases described below can be carried out using simulations,
>>          e.g. based on [ns-2] or [ns-3].  When feasible, it is also encouraged
>>          to perform testbed-based evaluations using Wi-Fi access points and
>>          endpoints running up-to-date IEEE 802.11 protocols, such as 802.11ac
>>          and the emerging Wi-Fi 6, to verify the viability of the candidate
>>          schemes.
>>  
>>     * "less minor"
>>        - Page 6. Section 3.1.1. Please do not define the bandwidth as 
>>     "infinite". That is problematic in a simulation. Rather, choose a big 
>>     enough value (e.g., 10 Gbps) that is big enough so that it will not 
>>     produce any congestion.
>>  
>> [Authors]  Thanks for raising this point.  The text in that section is now revised as:
>>  
>>   ... The fixed user is connected to the Internet via wired connection
>>   with sufficiently high bandwidth, for instance, 10 Gbps, so that the
>>   system is resource-limited on the wireless interface.
>>  
>>        - Page 10. Last bullet. I would use the expression "non-congestion 
>>     related losses" in this bullet, to be consistent with the cellular section.
>>  
>>  
>> [Authors]  We've discussed offline regarding this topic.  In order to describe
>> the nature of packet losses over Wi-Fi more accurately, the wording in the 3rd
>> bullet point of Sec. 4 has been revised to:
>>  
>>   * Packet transmissions over Wi-Fi are susceptible to contentions and
>>          collisions over the air.  Consequently, traffic load beyond a
>>          certain utilization level over a Wi-Fi network can introduce
>>   frequent collisions over the air and significant network overhead,
>>          as well as packet drops due to buffer overflow at the transmitters.
>>   This, in turn, leads to excessive delay, retransmissions, packet losses
>>   and lower effective bandwidth for applications.  Note, however, that
>>   the consequent delay and loss patterns caused by collisions are
>>   qualitatively different from those induced by congestion over a wired
>>   connection.
>>  
>>        - Page 11. Last paragraph. I wonder why the cellular section 
>>     recommends or suggests that the tests should be carried out only in a 
>>     simulator, whereas the WiFi section recommends to also run some 
>>     "evaluation" in a real testbed.
>>  
>>  
>> [Authors] Thanks for raising this important issue. Even though it is possible to carry out tests
>> over LTE (and 5G) networks, and actually it is already done so today, these tests cannot in the
>> general case be carried out in a deterministic fashion or to ensure repeatability, as these network
>> are in the control of cellular operators and there can various amounts of competing traffic in the
>> same cell(s). In practice, it is only in underground mines that one can carry out near deterministic
>> testing. Even there, it is not guaranteed either as workers in the mines carry with them their
>> iPhones and Androids.  Also, the underground mining setting may not reflect typical usage
>> patterns of a urban setting.
>>  
>> [Authors] The main reason for the lack of suggestion for testbed-based evaluations for cellular
>> in the draft is out of the above concerns of practical feasibility,  whereas Wi-Fi equipment is
>> relatively lower cost to acquire and setup.  Based on the reviewer comments, we've revised the
>> discussion in this paragraph (Sec. 4) to tune down the level of recommendation for testbed-based
>> Wi-Fi evaluations to "encourage":
>>  
>>   All test cases described below can be carried out using simulations,
>>          e.g. based on [ns-2] or [ns-3].  When feasible, it is also encouraged
>>          to perform testbed-based evaluations using Wi-Fi access points and
>>          endpoints running up-to-date IEEE 802.11 protocols, such as 802.11ac
>>          and the emerging Wi-Fi 6, to verify the viability of the candidate schemes.
> [SM] Thanks for the explanation. I find it valuable. Please consider explaining this in the draft, e.g., by squeezing a few sentences somewhere in the cellular section (suggestion: at the end of page 5).
> 
>>  
>>        - Page 16. "Congestion control" bullet includes the acronym [TBD]. 
>>     Does it mean "to be decided"? If it does, what is the decision? BTW, why 
>>     aren't TCP flows used in the cellular TCs (or did I miss them?)
>>  
>>  
>>  
>> [Authors] Good catch. This was indeed a "to-be-decided" reminder to the editors.
>> We've updated the reference for the default congestion control scheme of TCP to
>> [RFC5681] so as to align with the eval-test draft
>> (https://tools.ietf.org/html/draft-ietf-rmcat-eval-test-10 <https://tools.ietf.org/html/draft-ietf-rmcat-eval-test-10>, see Sec. 5.6 and 5.7).
>>  
>> The draft now specifies background FTP traffic over TCP (using default congestion
>> control per RFC5681) in both Sections 3.1 and 3.2 for cellular test cases.
>>  
>>     * minor/editorial
>>        - Abstract. "These applications are typically required to implement 
>>     congestion control". Grammar is correct, but somewhat ambiguous on what 
>>     requires what. Suggestion: "A congestion control algorithm is typically 
>>     required by these applications"
>>        - Page 4. 1st paragraph. "Besides, there are certain characteristics 
>>     which make the cellular network different and challenging than other 
>>     types of access network such as Wi-Fi and wired network”. Please fix 
>>     grammar near the word "than".
>>        - Page 5. "The key factors to define test cases for cellular network 
>>     are" should read either "for a cellular network", or "for cellular 
>>     networks". There are other occurrences of this further down in the text
>>        - Page 6. In my opinion, "there should be enough amount of time" 
>>     could be shortened to "there should be enough time"
>>        - Page 8. "intercity" --> "intensity" (?)
>>        - Page 11. First paragraph "singal" --> "signal"
>>        - Page 11. "Throughout this draft". I would rather say "Throughout 
>>     the 'WiFi networks' section" or "Throughout this section" something 
>>     similar, to avoid confusion with the cellular section
>>        - Page 12. 1st paragraph. "set up" --> "setup"
>>        - All over. Search and replace: "traffics" --> "traffic"
>>        - All over. Search and replace "followings" --> "following"
>>  
>> [Authors]  Thanks for identifying all these nits via your "minor/editorial" comments. The
>> draft has been updated accordingly.
>>  
>>     
>>     I hope this will help improve the quality of the text.
>>     
>>     Sergio
>>     
>>     
>>     On 15/01/19 00:24, Colin Perkins wrote:
>>     > This is to announce a working group last call on the “Evaluation Test Cases for Interactive Real-Time Media over Wireless Networks” (draft-ietf-rmcat-wireless-tests-06).
>>     >
>>     > Please send any final comments to the working group mailing list and the authors by 1 February 2019. If no substantive comments are received by that time, we intend to submit this draft to the IESG for publication as an Informational RFC.
>>     >
>>     > Colin
>>     > (as WG co-chair)
>>     >
>>     >
>>     >
>>     >
>>     >



-- 
Colin Perkins
https://csperkins.org/