Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)

[Mirja Kuehlewind <[e-mail]>]

Hi Roman,

Just on your discuss point below.

> On 4. Mar 2020, at 16:16, Roman Danyliw via Datatracker <[e-mail]> wrote:
> 
> Roman Danyliw has entered the following ballot position for
> draft-ietf-rmcat-wireless-tests-09: Discuss
> 
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-rmcat-wireless-tests/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> Please let me know if I've misunderstood the test execution protocol
> incorrectly:
> 
> Section 6.  Per the paragraph “The evaluation of the test cases are intended to
> carry out in a controlled lab environment … It is important to take appropriate
> caution to avoid leaking non-responsive traffic from unproven congestion
> avoidance techniques onto the open Internet”, this is good guidance in general
> case.  However, in the case of this document how applicable is it?  Didn’t
> Section 3 (“We, therefore, recommend that a cellular network simulator is used
> for the test cases defined in this document …” and practically establish it
> can’t be done without simulation with the scenario of the underground mine) and
> Section 4 (“We recommend to carry out the test cases as defined in this
> document using a simulator, such as [NS-2] or [NS-3]).   If all the testing is
> supposed to be in a simulator how is it leaking out onto the internet?  As far
> as I can tell, this helpful text is common in RMCAT document, but in this case
> could it please be tailored for the proposed testing regime.
> 
> Perhaps something on the order adding text on the order of “Given the
> difficulty of deterministic wireless testing, it is RECOMMENDED and expected
> that the tests described in this document would be done via simulation. 
> However, <in the case of not doing it that way> <leave the existing language>”

I would say that the proposed test cases are rather independent of the method used for evaluation: this could be simulation, emulation (where you plug a part of the simulated network into a real network), or testbed-based. Yes the author recommend simulation because having a real system would be really complicated (but not impossible…) and not necessarily easily provide reproducible results. However, as you note correctly there are no issues with potentially overloading a real network when you use simulations, that's why this case is not discussed in the security considerations section.

I guess we could clarify this in the text, however, note that this is the same text we also have in draft-ietf-rmcat-eval-test which is waiting in the RFC editor queue for this document.

Mirja


> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> ** Section 3.1.  Per “In this test case, each of the user/UE in the media
> session is an RMCAT compliant endpoint”, what is a “RMCAT compliant endpoint”? 
> Could this be cited please.
> 
> ** Section 3.1.  Per “At the beginning of the simulation, there should be
> enough time to warm-up the network”, intuitively the notion of “warm[ing]-up
> the network” makes sense.  However, is more precision required to side-by-side
> analysis of test runs of when the network is “warm enough”?
> 
> ** Section 4.  Per “Statistics collected from enterprise Wi-Fi networks show
> that the two dominant physical modes are 802.11n and 802.11ac, accounting for
> 41% and 58% of connected devices”, it is would be valuable to cite this value
> and provide a timestamp.  This distribution will certainly change as this
> document ages.
> 
> ** Section 4.  Per “Unless otherwise mentioned, test cases in this section are
> described using the underlying PHY- and MAC-layer parameters based on the IEEE
> 802.11n Standard.”, this focus on only 802.11n is surprising, since the next
> sentence establishes that 802.11.n is already less than half (41%) of the Wi-Fi
> traffic (and likely will continue to shrink).  Why not ac?
> 
> ** There appear to be some differences between the description of the Cellular
> (Section 3) and Wifi (Section 4) test. -- Section 4.1.2 and 4.2.2 are called
> “Test setup”, but Section 3.1.2 and 3.2.2. are called “Simulation setup”.  Was
> this intentional?
> 
> -- Section 4.*.4 discusses the expected test behavior, but Section 3.* does
> not?  Was that explicit?
> 
> 
> 
>