Re: [bmwg] Spencer Dawkins' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT) Tue, 17 April 2018 16:37 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id D0D1112706D; Tue, 17 Apr 2018 09:37:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 570ctbT3ft-e; Tue, 17 Apr 2018 09:37:21 -0700 (PDT)
Received: from ( []) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 44A23124B18; Tue, 17 Apr 2018 09:37:20 -0700 (PDT)
Received: from (unknown []) by (Postfix) with ESMTP id 9494B120042; Tue, 17 Apr 2018 22:06:53 +0530 (IST)
Received: from ( []) by (Postfix) with ESMTP id 1B78DFF91F; Tue, 17 Apr 2018 22:07:10 +0530 (IST)
Mime-Version: 1.0
X-Intloopheader: 0
Date: Tue, 17 Apr 2018 16:37:09 +0000
Content-Type: multipart/alternative; boundary="----=_Part_590_275800516.1523983029"
Message-ID: <>
X-Mailer: AfterLogic webmail client
To: "Spencer Dawkins at IETF" <>
Cc: "The IESG" <>,,, "Al Morton" <>,
In-Reply-To: <>
References: <> <>
X-Priority: 3 (Normal)
X-SMTP30-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: 1B78DFF91F.A9554
X-SMTP30-MailScanner: Found to be clean
X-Cloudmilter-Processed: 1
Archived-At: <>
Subject: Re: [bmwg] Spencer Dawkins' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 17 Apr 2018 16:37:25 -0000

Hi Spencer,
Thank you for reviewing the draft and providing your comments. Please find my responses inline with tag [Bhuvan]
Best regards,

On Tue, Apr 17, 2018 at 01:38 AM, Spencer Dawkins at IETF  wrote: "Text is hard" ... please see correction below.
On Mon, Apr 16, 2018 at 2:46 PM, Spencer Dawkins  wrote:
Spencer Dawkins has entered the following ballot position for
 draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection

 When responding, please keep the subject line intact and reply to all
 email addresses included in the To and CC lines. (Feel free to cut this
 introductory paragraph, however.)
 Please refer to (
 for more information about IESG DISCUSS and COMMENT positions.
 The document, along with other ballot positions, can be found here: (

 I have a few questions, at the No Objection level ... do the right thing, of

 I apologize for attempting to play amateur statistician, but it seems to me
 that this text

 4.7. Test Repeatability

    To increase the confidence in measured result, it is recommended
    that each test SHOULD be repeated a minimum of 10 times.

 is recommending a heuristic, when I'd think that you'd want to repeat a test
 until the results seem to be converging on some measure of central tendency,
 given some acceptable margin of error, and this text


    1. Establish the network connections between controller and network
    2. Query the controller for the discovered network topology
      information and compare it with the deployed network topology
    3. If the comparison is successful, increase the number of nodes by 1
      and repeat the trial.
      If the comparison is unsuccessful, decrease the number of nodes by
      1 and repeat the trial.
    4. Continue the trial until the comparison of step 3 is successful.
    5. Record the number of nodes for the last trial (Ns) where the
      topology comparison was successful.

 seems to beg for a binary search, especially if you're testing whether a
 controller can support a large number of controllers ...
"Can support a large number of NODES" ... in all caps, so I remember it ... 
[Bhuvan] I would like to clarify that the above procedure is for single test trial. We recommend to repeat the procedure for atleast 10 times for better accuracy of results. I think you got confused with the term trial in Step 3 of the procedure. If so, we will fix it in the new version. 
 This text

 Reference Test Setup:

    The test SHOULD use one of the test setups described in section 3.1
    or section 3.2 of this document in combination with Appendix A..

 or some variation is repeated about 16 times, and I'm not understanding why
 this is using BCP 14 language, and if BCP 14 language is the right thing to do,
 I'm not understanding why it's always SHOULD.

 I get the part that this will help compare results, if two researchers are
 running the same tests. Is there more to the requirement than that?
[Bhuvan] Our intention is to help compare results, if two testers are running  thesame tests. We do not have any other requirments than this..
 In this text,


    1. Perform the listed tests and launch a DoS attack towards
      controller while the trial is running.


     DoS attacks can be launched on one of the following interfaces.

      a. Northbound (e.g., Query for flow entries continuously on
        northbound interface)
      b. Management (e.g., Ping requests to controller's management
      c. Southbound (e.g., TCP SYN messages on southbound interface)

 is there a canonical description of "DoS attack" that researchers should be
 using, in order to compare results? These are just examples, right? 
[Bhuvan] You are correct. Note section is to give some examples to simulate DoS attacks 
 Is the choice of

   [OpenFlow Switch Specification]  ONF,"OpenFlow Switch Specification"
               Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.

 intentional? I'm googling that the current version of OpenFlow is 1.5.1, from
[Bhuvan] This is intentional as all our examples are derived based on this version of the specification.

DISCLAIMER: Privileged and/or Confidential information may be
contained in this message. If you are not the addressee of this message,
you may not copy, use or deliver this message to anyone. In such
event,you should destroy the message and kindly notify the sender by
reply e-mail.
It is understood that opinions or conclusions that do not relate to the
official business of the company are neither given nor endorsed by the