Re: [bmwg] Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

Benjamin Kaduk <> Thu, 19 April 2018 16:42 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id D042112E056; Thu, 19 Apr 2018 09:42:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -4.202
X-Spam-Status: No, score=-4.202 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id F1xbXFLrAJ7t; Thu, 19 Apr 2018 09:42:08 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 13A5D12E868; Thu, 19 Apr 2018 09:42:05 -0700 (PDT)
X-AuditID: 12074422-767ff700000029f0-ae-5ad8c6da1fb0
Received: from ( []) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 75.C3.10736.BD6C8DA5; Thu, 19 Apr 2018 12:42:04 -0400 (EDT)
Received: from (OUTGOING-AUTH-1.MIT.EDU []) by (8.13.8/8.9.2) with ESMTP id w3JGfwWM019021; Thu, 19 Apr 2018 12:41:59 -0400
Received: from ( []) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by (8.13.8/8.12.4) with ESMTP id w3JGfr2J027403 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 19 Apr 2018 12:41:55 -0400
Date: Thu, 19 Apr 2018 11:41:53 -0500
From: Benjamin Kaduk <>
To: Sarah B <>
Cc: The IESG <>,,, ALFRED MORTON <>,, "Bhuvan (Veryx Technologies)" <>
Message-ID: <>
References: <> <>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <>
User-Agent: Mutt/1.9.1 (2017-09-22)
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupjleLIzCtJLcpLzFFi42IRYrdT0b1z7EaUwZ4NkhZbj01ktPh++xWb xZej99gs+r/eZLN4cnUts8WMPxOZLTb/vcjowO7xsn8Oo8f3za3sHkuW/GTyeHRYKYAlissm JTUnsyy1SN8ugSvj9sKZzAXTxSqmnm9jbmB8KdjFyMEhIWAi0b7FoYuRi0NIYDGTxO2evywQ zkZGibbJXUwQzlUmiWefZjB2MXJysAioSjyadBHMZhNQkWjovswMYosIKElMWTmLHaSBWaCB SWLeg2awhLBAucS0VRtYQWxeoHVz/y+HWtHCKLHyzh4WiISgxMmZT8BsZgEtiRv/XjKB3Mcs IC2x/B8HSJhTwFHiwYrjYDNFBZQl9vYdYp/AKDALSfcsJN2zELoXMDKvYpRNya3SzU3MzClO TdYtTk7My0st0jXVy80s0UtNKd3ECA52F6UdjBP/eR1iFOBgVOLh/bDuRpQQa2JZcWXuIUZJ DiYlUd5jk4FCfEn5KZUZicUZ8UWlOanFhxglOJiVRHi924FyvCmJlVWpRfkwKWkOFiVx3sX7 90YJCaQnlqRmp6YWpBbBZGU4OJQkeJ8dBWoULEpNT61Iy8wpQUgzcXCCDOcBGv4fpIa3uCAx tzgzHSJ/ilFRSpx3NkhCACSRUZoH1wtKRhLZ+2teMYoDvSLMewykigeYyOC6XwENZgIabKAC NrgkESEl1cBYbrbKsEts4t6y5q5fgZzKGx8Zlv+ZvTpnywczmeMzd2QwnnOxDmfNLGdfwl2o NN/F91O2spju2xsCKv+0jZJqj9trGJi8T3j4tPaxDGv0J3+Ja+077q5MKkvcW3aiMTr7+5ES o4gH+ssnvdu9eO7ktzmT56p0zL3ooLH21HyL+8furvZc6jNDiaU4I9FQi7moOBEAvmZPfSED AAA=
Archived-At: <>
Subject: Re: [bmwg] Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 19 Apr 2018 16:42:18 -0000

On Thu, Apr 19, 2018 at 09:32:38AM -0700, Sarah B wrote:
> Hi Benjamin,
> 	Thanks for your review. A few comments inline below with SB//
> Thanks
> Sarah
> > 
> > ----------------------------------------------------------------------
> > ----------------------------------------------------------------------
> > 
> > In the Abstract:
> > 
> >   This document defines the methodologies for benchmarking control
> >   plane performance of SDN controllers.
> > 
> > Why "the" methodologies?   That seems more authoritative than is
> > appropriate in an Informational document.
> > 
> Sure, we can remove this.

(I probably should have explicitly said that "a methodology" is

> > 
> > Thank you for adding consideration to key distribution in Section
> > 4.4, as noted by the secdir review.  But insisting on having key
> > distribution done prior to testing gives the impression that keys
> > are distributed once and updated never, which has questionable
> > security properties.  Perhaps there is value in doing some testing
> > while rekeyeing is in progress?
> We'll discuss internally.. the goal here wasn't to test rekeying, which is why the test doesn't call for anything other than doing the distribution prior to the test, and remember, this is lab testing; this isn't what happens on a live network, but it's an interesting point, and we'll discuss.

Okay.  I'm just approaching this from a "social norms" perspective
(regarding expectations that re-keying will be done), but I
definitely understand that there is less need for benchmarking
rekeying performance than the cases already covered.

> > 
> > I agree with others that the statistical methodology is not clearly
> > justified, such as the sample size of 10 in Section 4.7 (with no
> > consideration for sample relative variance), use of sample vs.
> > population veriance, etc.
> As a draft, we aren't saying you only have to do 10; it's been a point of discussion in BMWG for some time, what number do you put? We haven't always found a happy medium to this. We're suggesting you perform the test 10 times, but there's nothing that stops a tester from making that <n> times; for example, 100 times. It's important to note how many times the test was performed and move on. We can clarify this text to reflect that, if it helps. However, note, BMWG is about repeatability, and that was the goal here.

Well, there's a huge amount of literature (and, e.g., textbooks) on
statistical anlaysis and how to decide that a series of measurements
has converged on a stable result.  So it might be better to just say
that benchmarkers must ensure that the data has converged and note
that this will require running many repeated trials, but not
necessarily list a specific value.