Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)

Mirja Kuehlewind <> Wed, 04 March 2020 15:50 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 09F9E3A1187; Wed, 4 Mar 2020 07:50:37 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id dPnWgKp91Btn; Wed, 4 Mar 2020 07:50:33 -0800 (PST)
Received: from ( [IPv6:2a01:488:42:1000:50ed:8223::]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 4FA9F3A1152; Wed, 4 Mar 2020 07:50:30 -0800 (PST)
Received: from ([2003:de:e723:9a00:8480:9b28:d0f2:2131]); authenticated by running ExIM with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1j9WHr-0006LL-1l; Wed, 04 Mar 2020 16:50:19 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Mirja Kuehlewind <>
In-Reply-To: <>
Date: Wed, 4 Mar 2020 16:50:18 +0100
Cc: The IESG <>,,,,
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <>
To: Roman Danyliw <>
X-Mailer: Apple Mail (2.3445.104.11)
X-HE-SMSGID: 1j9WHr-0006LL-1l
Archived-At: <>
Subject: Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "RTP Media Congestion Avoidance Techniques \(RMCAT\) Working Group discussion list." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 04 Mar 2020 15:50:38 -0000

Hi Roman,

Just on your discuss point below.

> On 4. Mar 2020, at 16:16, Roman Danyliw via Datatracker <> wrote:
> Roman Danyliw has entered the following ballot position for
> draft-ietf-rmcat-wireless-tests-09: Discuss
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> Please refer to
> for more information about IESG DISCUSS and COMMENT positions.
> The document, along with other ballot positions, can be found here:
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> Please let me know if I've misunderstood the test execution protocol
> incorrectly:
> Section 6.  Per the paragraph “The evaluation of the test cases are intended to
> carry out in a controlled lab environment … It is important to take appropriate
> caution to avoid leaking non-responsive traffic from unproven congestion
> avoidance techniques onto the open Internet”, this is good guidance in general
> case.  However, in the case of this document how applicable is it?  Didn’t
> Section 3 (“We, therefore, recommend that a cellular network simulator is used
> for the test cases defined in this document …” and practically establish it
> can’t be done without simulation with the scenario of the underground mine) and
> Section 4 (“We recommend to carry out the test cases as defined in this
> document using a simulator, such as [NS-2] or [NS-3]).   If all the testing is
> supposed to be in a simulator how is it leaking out onto the internet?  As far
> as I can tell, this helpful text is common in RMCAT document, but in this case
> could it please be tailored for the proposed testing regime.
> Perhaps something on the order adding text on the order of “Given the
> difficulty of deterministic wireless testing, it is RECOMMENDED and expected
> that the tests described in this document would be done via simulation. 
> However, <in the case of not doing it that way> <leave the existing language>”

I would say that the proposed test cases are rather independent of the method used for evaluation: this could be simulation, emulation (where you plug a part of the simulated network into a real network), or testbed-based. Yes the author recommend simulation because having a real system would be really complicated (but not impossible…) and not necessarily easily provide reproducible results. However, as you note correctly there are no issues with potentially overloading a real network when you use simulations, that's why this case is not discussed in the security considerations section.

I guess we could clarify this in the text, however, note that this is the same text we also have in draft-ietf-rmcat-eval-test which is waiting in the RFC editor queue for this document.


> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> ** Section 3.1.  Per “In this test case, each of the user/UE in the media
> session is an RMCAT compliant endpoint”, what is a “RMCAT compliant endpoint”? 
> Could this be cited please.
> ** Section 3.1.  Per “At the beginning of the simulation, there should be
> enough time to warm-up the network”, intuitively the notion of “warm[ing]-up
> the network” makes sense.  However, is more precision required to side-by-side
> analysis of test runs of when the network is “warm enough”?
> ** Section 4.  Per “Statistics collected from enterprise Wi-Fi networks show
> that the two dominant physical modes are 802.11n and 802.11ac, accounting for
> 41% and 58% of connected devices”, it is would be valuable to cite this value
> and provide a timestamp.  This distribution will certainly change as this
> document ages.
> ** Section 4.  Per “Unless otherwise mentioned, test cases in this section are
> described using the underlying PHY- and MAC-layer parameters based on the IEEE
> 802.11n Standard.”, this focus on only 802.11n is surprising, since the next
> sentence establishes that 802.11.n is already less than half (41%) of the Wi-Fi
> traffic (and likely will continue to shrink).  Why not ac?
> ** There appear to be some differences between the description of the Cellular
> (Section 3) and Wifi (Section 4) test. -- Section 4.1.2 and 4.2.2 are called
> “Test setup”, but Section 3.1.2 and 3.2.2. are called “Simulation setup”.  Was
> this intentional?
> -- Section 4.*.4 discusses the expected test behavior, but Section 3.* does
> not?  Was that explicit?