Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)

Mirja Kuehlewind <ietf@kuehlewind.net> Wed, 04 March 2020 15:50 UTC

Return-Path: <ietf@kuehlewind.net>
X-Original-To: rmcat@ietfa.amsl.com
Delivered-To: rmcat@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 09F9E3A1187; Wed, 4 Mar 2020 07:50:37 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level:
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dPnWgKp91Btn; Wed, 4 Mar 2020 07:50:33 -0800 (PST)
Received: from wp513.webpack.hosteurope.de (wp513.webpack.hosteurope.de [IPv6:2a01:488:42:1000:50ed:8223::]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4FA9F3A1152; Wed, 4 Mar 2020 07:50:30 -0800 (PST)
Received: from p200300dee7239a0084809b28d0f22131.dip0.t-ipconnect.de ([2003:de:e723:9a00:8480:9b28:d0f2:2131]); authenticated by wp513.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1j9WHr-0006LL-1l; Wed, 04 Mar 2020 16:50:19 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Mirja Kuehlewind <ietf@kuehlewind.net>
In-Reply-To: <158333496772.29337.12478553941060973013@ietfa.amsl.com>
Date: Wed, 4 Mar 2020 16:50:18 +0100
Cc: The IESG <iesg@ietf.org>, rmcat-chairs@ietf.org, draft-ietf-rmcat-wireless-tests@ietf.org, csp@csperkins.org, rmcat@ietf.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <131F5662-796B-4664-95F0-22A3A0118117@kuehlewind.net>
References: <158333496772.29337.12478553941060973013@ietfa.amsl.com>
To: Roman Danyliw <rdd@cert.org>
X-Mailer: Apple Mail (2.3445.104.11)
X-bounce-key: webpack.hosteurope.de;ietf@kuehlewind.net;1583337033;199901ee;
X-HE-SMSGID: 1j9WHr-0006LL-1l
Archived-At: <https://mailarchive.ietf.org/arch/msg/rmcat/RmJnfTR2YcZiQS7E-ig18JPJeRg>
Subject: Re: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)
X-BeenThere: rmcat@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "RTP Media Congestion Avoidance Techniques \(RMCAT\) Working Group discussion list." <rmcat.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rmcat>, <mailto:rmcat-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/rmcat/>
List-Post: <mailto:rmcat@ietf.org>
List-Help: <mailto:rmcat-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rmcat>, <mailto:rmcat-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Mar 2020 15:50:38 -0000

Hi Roman,

Just on your discuss point below.

> On 4. Mar 2020, at 16:16, Roman Danyliw via Datatracker <noreply@ietf.org> wrote:
> 
> Roman Danyliw has entered the following ballot position for
> draft-ietf-rmcat-wireless-tests-09: Discuss
> 
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-rmcat-wireless-tests/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> Please let me know if I've misunderstood the test execution protocol
> incorrectly:
> 
> Section 6.  Per the paragraph “The evaluation of the test cases are intended to
> carry out in a controlled lab environment … It is important to take appropriate
> caution to avoid leaking non-responsive traffic from unproven congestion
> avoidance techniques onto the open Internet”, this is good guidance in general
> case.  However, in the case of this document how applicable is it?  Didn’t
> Section 3 (“We, therefore, recommend that a cellular network simulator is used
> for the test cases defined in this document …” and practically establish it
> can’t be done without simulation with the scenario of the underground mine) and
> Section 4 (“We recommend to carry out the test cases as defined in this
> document using a simulator, such as [NS-2] or [NS-3]).   If all the testing is
> supposed to be in a simulator how is it leaking out onto the internet?  As far
> as I can tell, this helpful text is common in RMCAT document, but in this case
> could it please be tailored for the proposed testing regime.
> 
> Perhaps something on the order adding text on the order of “Given the
> difficulty of deterministic wireless testing, it is RECOMMENDED and expected
> that the tests described in this document would be done via simulation. 
> However, <in the case of not doing it that way> <leave the existing language>”

I would say that the proposed test cases are rather independent of the method used for evaluation: this could be simulation, emulation (where you plug a part of the simulated network into a real network), or testbed-based. Yes the author recommend simulation because having a real system would be really complicated (but not impossible…) and not necessarily easily provide reproducible results. However, as you note correctly there are no issues with potentially overloading a real network when you use simulations, that's why this case is not discussed in the security considerations section.

I guess we could clarify this in the text, however, note that this is the same text we also have in draft-ietf-rmcat-eval-test which is waiting in the RFC editor queue for this document.

Mirja


> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> ** Section 3.1.  Per “In this test case, each of the user/UE in the media
> session is an RMCAT compliant endpoint”, what is a “RMCAT compliant endpoint”? 
> Could this be cited please.
> 
> ** Section 3.1.  Per “At the beginning of the simulation, there should be
> enough time to warm-up the network”, intuitively the notion of “warm[ing]-up
> the network” makes sense.  However, is more precision required to side-by-side
> analysis of test runs of when the network is “warm enough”?
> 
> ** Section 4.  Per “Statistics collected from enterprise Wi-Fi networks show
> that the two dominant physical modes are 802.11n and 802.11ac, accounting for
> 41% and 58% of connected devices”, it is would be valuable to cite this value
> and provide a timestamp.  This distribution will certainly change as this
> document ages.
> 
> ** Section 4.  Per “Unless otherwise mentioned, test cases in this section are
> described using the underlying PHY- and MAC-layer parameters based on the IEEE
> 802.11n Standard.”, this focus on only 802.11n is surprising, since the next
> sentence establishes that 802.11.n is already less than half (41%) of the Wi-Fi
> traffic (and likely will continue to shrink).  Why not ac?
> 
> ** There appear to be some differences between the description of the Cellular
> (Section 3) and Wifi (Section 4) test. -- Section 4.1.2 and 4.2.2 are called
> “Test setup”, but Section 3.1.2 and 3.2.2. are called “Simulation setup”.  Was
> this intentional?
> 
> -- Section 4.*.4 discusses the expected test behavior, but Section 3.* does
> not?  Was that explicit?
> 
> 
> 
>