Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)

["Xiaoqing Zhu (xiaoqzhu)" <[e-mail]>]

Hi Roman, 

Many thanks for your thorough review of this draft.  We have updated it to version -11: 
https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-wireless-tests-11

Please also see our detailed responses, tagged [authors] inline below. 

Best regards,
Xiaoqing (on behalf of all authors)

On 3/4/20, 9:16 AM, "Roman Danyliw via Datatracker" <[e-mail]> wrote:

    Roman Danyliw has entered the following ballot position for
    draft-ietf-rmcat-wireless-tests-09: Discuss
    
    When responding, please keep the subject line intact and reply to all
    email addresses included in the To and CC lines. (Feel free to cut this
    introductory paragraph, however.)
    
    
    Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
    for more information about IESG DISCUSS and COMMENT positions.
    
    
    The document, along with other ballot positions, can be found here:
    https://datatracker.ietf.org/doc/draft-ietf-rmcat-wireless-tests/
    
    
    
    ----------------------------------------------------------------------
    DISCUSS:
    ----------------------------------------------------------------------
    
    Please let me know if I've misunderstood the test execution protocol
    incorrectly:
    
    Section 6.  Per the paragraph “The evaluation of the test cases are intended to
    carry out in a controlled lab environment … It is important to take appropriate
    caution to avoid leaking non-responsive traffic from unproven congestion
    avoidance techniques onto the open Internet”, this is good guidance in general
    case.  However, in the case of this document how applicable is it?  Didn’t
    Section 3 (“We, therefore, recommend that a cellular network simulator is used
    for the test cases defined in this document …” and practically establish it
    can’t be done without simulation with the scenario of the underground mine) and
    Section 4 (“We recommend to carry out the test cases as defined in this
    document using a simulator, such as [NS-2] or [NS-3]).   If all the testing is
    supposed to be in a simulator how is it leaking out onto the internet?  As far
    as I can tell, this helpful text is common in RMCAT document, but in this case
    could it please be tailored for the proposed testing regime.
    
    Perhaps something on the order adding text on the order of “Given the
    difficulty of deterministic wireless testing, it is RECOMMENDED and expected
    that the tests described in this document would be done via simulation. 
    However, <in the case of not doing it that way> <leave the existing language>”

[authors] Thanks for the suggestion.  We have now revised the paragraph (now in Sec. 5)
to begin with: 

	Given the difficulty of deterministic wireless testing, it is
   	recommended and expected that the tests described in this document
	 would be done via simulations.  However, in the case where these test
   	cases are carried out in a testbed setting, the evaluation should
   	take place in a controlled lab environment.
    
   
    ----------------------------------------------------------------------
    COMMENT:
    ----------------------------------------------------------------------
    
    ** Section 3.1.  Per “In this test case, each of the user/UE in the media
    session is an RMCAT compliant endpoint”, what is a “RMCAT compliant endpoint”? 
    Could this be cited please.
   
[authors] We realize that the term RMCAT will no longer be relevant after all works of the 
WG are completed. We have therefore revised the sentence (now in Sec. 2.1 and 2.2) to:

	In this test case, each user/UE in the media session is an endpoint following RTP-based
   	congestion control.
 
    ** Section 3.1.  Per “At the beginning of the simulation, there should be
    enough time to warm-up the network”, intuitively the notion of “warm[ing]-up
    the network” makes sense.  However, is more precision required to side-by-side
    analysis of test runs of when the network is “warm enough”?

[authors] Good point. In the test case, the evaluation time is specified to be 30 seconds
after the start of the simulation. We've added the following as clarifications to the beginning
of both Sections 2.1 and 2.2 (previously 3.1 and 3.2):

	Typically, the evaluation period starts 30 seconds after test initialization.

    ** Section 4.  Per “Statistics collected from enterprise Wi-Fi networks show
    that the two dominant physical modes are 802.11n and 802.11ac, accounting for
    41% and 58% of connected devices”, it is would be valuable to cite this value
    and provide a timestamp.  This distribution will certainly change as this
    document ages.

[authors] These statistics are collected from our internal analytics dashboard
based from enterprise customer deployment. Unfortunately, we cannot provide
a public reference link to that data source. We have searched for alternative
sources of similar information (such as a white paper) but haven't yet found a
good substitute. If you are aware of any pointers that's very much welcome.


    
    ** Section 4.  Per “Unless otherwise mentioned, test cases in this section are
    described using the underlying PHY- and MAC-layer parameters based on the IEEE
    802.11n Standard.”, this focus on only 802.11n is surprising, since the next
    sentence establishes that 802.11.n is already less than half (41%) of the Wi-Fi
    traffic (and likely will continue to shrink).  Why not ac?

[authors] This draft was initially developed in 2015. At that time 11n was still the
dominant technology for endpoints and 11ac was not supported in ns-3.
We later updated the statistics on respective portions of 11n vs. 11ac clients in the draft.
On the other hand, since we haven't yet got the time/resource to update all ns3-based
test case implementations to be based on 11ac, our test case specifications are stuck
with the 11n mode.  Instead, we've added some text right after that sentence to encourage
practitioners to evaluate using up-to-date versions of Wi-Fi:

	As Wi-Fi standards evolve over time -- for instance, with the introduction of the
	emerging Wi-Fi 6 (based on IEEE 802.11ax) products -- the PHY- and MAC-layer
	test case specifications need to be updated accordingly to reflect such changes. 

    
    ** There appear to be some differences between the description of the Cellular
    (Section 3) and Wifi (Section 4) test. -- Section 4.1.2 and 4.2.2 are called
    “Test setup”, but Section 3.1.2 and 3.2.2. are called “Simulation setup”.  Was
    this intentional?

[authors] This is an artificat due to the merge of two separate drafts. Since
our validation efforts for cellular is in simulation only whereas for WiFi we've
implemented both simulation-based and testbed-based tests, we've updated the 
headings for Wi-Fi tests to be ' Test/simulation setup' instead. 


    
    -- Section 4.*.4 discusses the expected test behavior, but Section 3.* does
    not?  Was that explicit?
    
 [authors]. Again, this inconsistency in style is due to the draft merging. The draft originally
contained some inline texts on the expected behavior of cellular test cases.  Per your comment,
we have moved them out to be stand-along subsections (Sec. 2.1.3 and 2.2.3), which reads:

	The investigated congestion control algorithms should result in
   	maximum possible network utilization and stability in terms of rate
   	variations, lowest possible end to end frame latency, network latency
   	and Packet Loss Rate (PLR) at different cell load levels.