Re: [bmwg] Spencer Dawkins' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

[Spencer Dawkins at IETF <[e-mail]>]

"Text is hard" ... please see correction below.

On Mon, Apr 16, 2018 at 2:46 PM, Spencer Dawkins <
[e-mail]> wrote:

> Spencer Dawkins has entered the following ballot position for
> draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
>
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-
> controller-benchmark-meth/
>
>
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> I have a few questions, at the No Objection level ... do the right thing,
> of
> course.
>
> I apologize for attempting to play amateur statistician, but it seems to me
> that this text
>
> 4.7. Test Repeatability
>
>    To increase the confidence in measured result, it is recommended
>    that each test SHOULD be repeated a minimum of 10 times.
>
> is recommending a heuristic, when I'd think that you'd want to repeat a
> test
> until the results seem to be converging on some measure of central
> tendency,
> given some acceptable margin of error, and this text
>
> Procedure:
>
>    1. Establish the network connections between controller and network
>      nodes.
>    2. Query the controller for the discovered network topology
>      information and compare it with the deployed network topology
>      information.
>    3. If the comparison is successful, increase the number of nodes by 1
>      and repeat the trial.
>      If the comparison is unsuccessful, decrease the number of nodes by
>      1 and repeat the trial.
>    4. Continue the trial until the comparison of step 3 is successful.
>    5. Record the number of nodes for the last trial (Ns) where the
>      topology comparison was successful.
>
> seems to beg for a binary search, especially if you're testing whether a
> controller can support a large number of controllers ...
>

"Can support a large number of NODES" ... in all caps, so I remember it ...

Spencer


> This text
>
> Reference Test Setup:
>
>    The test SHOULD use one of the test setups described in section 3.1
>    or section 3.2 of this document in combination with Appendix A.
>
> or some variation is repeated about 16 times, and I'm not understanding why
> this is using BCP 14 language, and if BCP 14 language is the right thing
> to do,
> I'm not understanding why it's always SHOULD.
>
> I get the part that this will help compare results, if two researchers are
> running the same tests. Is there more to the requirement than that?
>
> In this text,
>
> Procedure:
>
>    1. Perform the listed tests and launch a DoS attack towards
>      controller while the trial is running.
>
>    Note:
>
>     DoS attacks can be launched on one of the following interfaces.
>
>      a. Northbound (e.g., Query for flow entries continuously on
>        northbound interface)
>      b. Management (e.g., Ping requests to controller's management
>        interface)
>      c. Southbound (e.g., TCP SYN messages on southbound interface)
>
> is there a canonical description of "DoS attack" that researchers should be
> using, in order to compare results? These are just examples, right?
>
> Is the choice of
>
>   [OpenFlow Switch Specification]  ONF,"OpenFlow Switch Specification"
>               Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.
>
> intentional? I'm googling that the current version of OpenFlow is 1.5.1,
> from
> 2015.
>
>
>