Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

["Banks, Sarah" <[e-mail]>]

Hi Brian,
	Let me address inline.

>
>Second, we must ensure that the collection of tests in the draft are
>appropriate. Tests 6.1.1 and 6.1.2 are inverse of one another and
>therefore redundant. We should consolidate them into a single family of
>controller-response tests and simplify.

SB// I don't know that I agree with this; inverse tests aren't redundant
in my opinion; I'm not sure where the value is in consolidating them into
a family (is it not your position that they're redundant? That would imply
removing one or the other..). I have no issue with them being separate
tests, in the same section, which they are. To my eye, that's perfectly
readable.

>Test
> 6.1.3 does not adequately isolate the controller in the test because the
>overall response time depends on the response time of the switches. If a
>tester runs the same test on the same controller with different switches,
>he or she will get different answers.
> The goal of a benchmark test is to enable apples-to-apples comparisons
>between controllers. I think a step back to reconsider the collection of
>proposed tests is in order.

SB// Perhaps this could be a variable called out; I tend to agree, if the
purpose is to test the controller and not the switches, this should be
documented as part of the topology at the "start of the hour", so that the
only equipment being tested is the controller.

>
>
>
>Third, we must consider the work currently being done by other standards
>bodies. The Open Networking Foundation Testing & Interop Working Group,
>for example, is currently developing it’s own OpenFlow Controller
>benchmarks. I would prefer a combined solution
> if possible.
>
>The draft under consideration should not go further until these issues
>are addressed. 

I partially agree, and I believe the authors might as well, in that we
know other bodies are working on OpenFlow, for example. I do think some
understanding of what's happening within those organizations would be an
excellent starting point. However, I don't believe that this issue, with
the 2 aforementioned issues, are a reason to stop proceeding with the
document. If, through the process of working through this, we determine as
a group, or the authors determine on their own, that what other groups are
doing is sufficient, we can make that call at that time.  It's very early
in the draft, and I wouldn't mind seeing where this takes us as a group,
particularly if we can have some form of communication or understanding
with other groups throughout the industry who might be looking at this
problem in some way.

Kind regards,
Sarah