Re: [bmwg] Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

[Sarah B <[e-mail]>]

Hi Benjamin,
	Thanks for your review. A few comments inline below with SB//

Thanks
Sarah
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> In the Abstract:
> 
>   This document defines the methodologies for benchmarking control
>   plane performance of SDN controllers.
> 
> Why "the" methodologies?   That seems more authoritative than is
> appropriate in an Informational document.
> 

Sure, we can remove this.

> 
> Thank you for adding consideration to key distribution in Section
> 4.4, as noted by the secdir review.  But insisting on having key
> distribution done prior to testing gives the impression that keys
> are distributed once and updated never, which has questionable
> security properties.  Perhaps there is value in doing some testing
> while rekeyeing is in progress?

We'll discuss internally.. the goal here wasn't to test rekeying, which is why the test doesn't call for anything other than doing the distribution prior to the test, and remember, this is lab testing; this isn't what happens on a live network, but it's an interesting point, and we'll discuss.

> 
> I agree with others that the statistical methodology is not clearly
> justified, such as the sample size of 10 in Section 4.7 (with no
> consideration for sample relative variance), use of sample vs.
> population veriance, etc.

As a draft, we aren't saying you only have to do 10; it's been a point of discussion in BMWG for some time, what number do you put? We haven't always found a happy medium to this. We're suggesting you perform the test 10 times, but there's nothing that stops a tester from making that <n> times; for example, 100 times. It's important to note how many times the test was performed and move on. We can clarify this text to reflect that, if it helps. However, note, BMWG is about repeatability, and that was the goal here.

> 
> It seems like the measurements being described sometimes start the
> timer at an event at a network element and other times start the
> timer when a message enters the SDN controller itself (similarly for
> outgoing messages), which seems to include a different treatment of
> propagation delays in the network, for different tests.  Assuming
> these differences were made by conscious choice, it might be nice to
> describe why the network propagation is/is not included for any
> given measurement.
> 
> It looks like the term "Nxrn" is introduced implicitly and the
> reader is supposed to infer that the 'n' represents a counter, with
> Nrx1 corresponding to the first measurement, Nrx2 the second, etc.
> It's probably worth mentioning this explicitly, for all fields that
> are measured on a per-trial/counter basis.
> 
SB// OK

> 
> Appendix B.3 indicates that plain TCP or TLS can be used for
> communications between switch and controller.  It seems like this
> would be a highly relevant test parameter to report with the results
> for the tests described in this document, since TLS would introduce
> additional overhead to be quantified!

SB// Agreed. The intent is to have the scenario specifically documented by the tester to reflect the choice made; we'll clarify the text.