Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

["Castelli, Brian" <[e-mail]>]

Thank you, Sarah.

The specific inverse example that I wrote about is a literal inverse with
considerable overlap between test cases. If I measure the rate, I can do
1/rate to get the time. If I measure the time, I can do 1/time to get the
rate. I see no point in putting a tester through both sets of tests in
their current form. Plus, as I wrote about in my previous set of comments,
there are places in the test cases where the tester is urged to repeat the
same test multiple times without changing test conditions and then plot
the result on a graph. This is an example of extra work that provides
little or no value to the tester. We can accomplish the goals of 6.1.1 and
6.1.2 with a test case or two that use 40% fewer words in the document.

My two main concerns are getting the terminology right and making sure
that the proposed set of tests will achieve our goals. I believe those
goals ought to include a common taxonomy, enabling apples-to-apples
comparisons, and minimizing the time/work required to execute the test
cases.

I am willing to help improve the benchmark. I understand that it is an
early draft. The time is right to make sure that we are developing the
best specification that we can.

How do we continue this process? Via email to this list? I am willing to
help.

On 6/19/14, 8:54 AM, "Banks, Sarah" <[e-mail]> wrote:

>Hi Brian,
>       Let me address inline.
>
>>
>>Second, we must ensure that the collection of tests in the draft are
>>appropriate. Tests 6.1.1 and 6.1.2 are inverse of one another and
>>therefore redundant. We should consolidate them into a single family of
>>controller-response tests and simplify.
>
>SB// I don't know that I agree with this; inverse tests aren't redundant
>in my opinion; I'm not sure where the value is in consolidating them into
>a family (is it not your position that they're redundant? That would imply
>removing one or the other..). I have no issue with them being separate
>tests, in the same section, which they are. To my eye, that's perfectly
>readable.
>
>>Test
>> 6.1.3 does not adequately isolate the controller in the test because the
>>overall response time depends on the response time of the switches. If a
>>tester runs the same test on the same controller with different switches,
>>he or she will get different answers.
>> The goal of a benchmark test is to enable apples-to-apples comparisons
>>between controllers. I think a step back to reconsider the collection of
>>proposed tests is in order.
>
>SB// Perhaps this could be a variable called out; I tend to agree, if the
>purpose is to test the controller and not the switches, this should be
>documented as part of the topology at the "start of the hour", so that the
>only equipment being tested is the controller.
>
>>
>>
>>
>>Third, we must consider the work currently being done by other standards
>>bodies. The Open Networking Foundation Testing & Interop Working Group,
>>for example, is currently developing it¹s own OpenFlow Controller
>>benchmarks. I would prefer a combined solution
>> if possible.
>>
>>The draft under consideration should not go further until these issues
>>are addressed.
>
>I partially agree, and I believe the authors might as well, in that we
>know other bodies are working on OpenFlow, for example. I do think some
>understanding of what's happening within those organizations would be an
>excellent starting point. However, I don't believe that this issue, with
>the 2 aforementioned issues, are a reason to stop proceeding with the
>document. If, through the process of working through this, we determine as
>a group, or the authors determine on their own, that what other groups are
>doing is sufficient, we can make that call at that time.  It's very early
>in the draft, and I wouldn't mind seeing where this takes us as a group,
>particularly if we can have some form of communication or understanding
>with other groups throughout the industry who might be looking at this
>problem in some way.
>
>Kind regards,
>Sarah
>

E-mail confidentiality.
--------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. [phone]
Fax No. [phone]

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US,

Spirent Communications,
26750 Agoura Road, Calabasas, CA, 91302, USA.
Tel No. 1-818-676- 2300