Re: [bmwg] Spencer Dawkins' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

[[e-mail]]

Hi Spencer,
Thank you for reviewing the draft and providing your comments. Please find my responses inline with tag [Bhuvan]
Best regards,
Bhuvan

On Tue, Apr 17, 2018 at 01:38 AM, Spencer Dawkins at IETF  wrote: "Text is hard" ... please see correction below.
On Mon, Apr 16, 2018 at 2:46 PM, Spencer Dawkins  wrote:
Spencer Dawkins has entered the following ballot position for
 draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection

 When responding, please keep the subject line intact and reply to all
 email addresses included in the To and CC lines. (Feel free to cut this
 introductory paragraph, however.)
 Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html (https://www.ietf.org/iesg/statement/discuss-criteria.html)
 for more information about IESG DISCUSS and COMMENT positions.
 The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/ (https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/)
 ----------------------------------------------------------------------
 COMMENT:
 ----------------------------------------------------------------------

 I have a few questions, at the No Objection level ... do the right thing, of
 course.

 I apologize for attempting to play amateur statistician, but it seems to me
 that this text

 4.7. Test Repeatability

    To increase the confidence in measured result, it is recommended
    that each test SHOULD be repeated a minimum of 10 times.

 is recommending a heuristic, when I'd think that you'd want to repeat a test
 until the results seem to be converging on some measure of central tendency,
 given some acceptable margin of error, and this text

 Procedure:

    1. Establish the network connections between controller and network
      nodes.
    2. Query the controller for the discovered network topology
      information and compare it with the deployed network topology
      information.
    3. If the comparison is successful, increase the number of nodes by 1
      and repeat the trial.
      If the comparison is unsuccessful, decrease the number of nodes by
      1 and repeat the trial.
    4. Continue the trial until the comparison of step 3 is successful.
    5. Record the number of nodes for the last trial (Ns) where the
      topology comparison was successful.

 seems to beg for a binary search, especially if you're testing whether a
 controller can support a large number of controllers ...
"Can support a large number of NODES" ... in all caps, so I remember it ... 
[Bhuvan] I would like to clarify that the above procedure is for single test trial. We recommend to repeat the procedure for atleast 10 times for better accuracy of results. I think you got confused with the term trial in Step 3 of the procedure. If so, we will fix it in the new version. 
Spencer
 This text

 Reference Test Setup:

    The test SHOULD use one of the test setups described in section 3.1
    or section 3.2 of this document in combination with Appendix A..

 or some variation is repeated about 16 times, and I'm not understanding why
 this is using BCP 14 language, and if BCP 14 language is the right thing to do,
 I'm not understanding why it's always SHOULD.

 I get the part that this will help compare results, if two researchers are
 running the same tests. Is there more to the requirement than that?
[Bhuvan] Our intention is to help compare results, if two testers are running  thesame tests. We do not have any other requirments than this..
 In this text,

 Procedure:

    1. Perform the listed tests and launch a DoS attack towards
      controller while the trial is running.

    Note:

     DoS attacks can be launched on one of the following interfaces.

      a. Northbound (e.g., Query for flow entries continuously on
        northbound interface)
      b. Management (e.g., Ping requests to controller's management
        interface)
      c. Southbound (e.g., TCP SYN messages on southbound interface)

 is there a canonical description of "DoS attack" that researchers should be
 using, in order to compare results? These are just examples, right? 
[Bhuvan] You are correct. Note section is to give some examples to simulate DoS attacks 
 Is the choice of

   [OpenFlow Switch Specification]  ONF,"OpenFlow Switch Specification"
               Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.

 intentional? I'm googling that the current version of OpenFlow is 1.5.1, from
 2015.
[Bhuvan] This is intentional as all our examples are derived based on this version of the specification.

DISCLAIMER: Privileged and/or Confidential information may be
contained in this message. If you are not the addressee of this message,
you may not copy, use or deliver this message to anyone. In such
event,you should destroy the message and kindly notify the sender by
reply e-mail.
It is understood that opinions or conclusions that do not relate to the
official business of the company are neither given nor endorsed by the
company.