Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

["Bhuvan \(Veryx Technologies\)" <[e-mail]>]

Hi Sarah,

Thanks for your response. I thought of providing similar response to Brain.

Brain,

Hope Sarah's response provides the required clarity. Please get back for additional clarifications and comments.

Best Regards,
Bhuvan

-----Original Message-----
From: Banks, Sarah [mailto:[e-mail]] 
Sent: Thursday, June 19, 2014 6:24 PM
To: Castelli, Brian; Bhuvan (Veryx Technologies); [e-mail]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Hi Brian,
	Let me address inline.

>
>Second, we must ensure that the collection of tests in the draft are 
>appropriate. Tests 6.1.1 and 6.1.2 are inverse of one another and 
>therefore redundant. We should consolidate them into a single family of 
>controller-response tests and simplify.

SB// I don't know that I agree with this; inverse tests aren't redundant in my opinion; I'm not sure where the value is in consolidating them into a family (is it not your position that they're redundant? That would imply removing one or the other..). I have no issue with them being separate tests, in the same section, which they are. To my eye, that's perfectly readable.

>Test
> 6.1.3 does not adequately isolate the controller in the test because 
>the overall response time depends on the response time of the switches. 
>If a tester runs the same test on the same controller with different 
>switches, he or she will get different answers.
> The goal of a benchmark test is to enable apples-to-apples comparisons 
>between controllers. I think a step back to reconsider the collection 
>of proposed tests is in order.

SB// Perhaps this could be a variable called out; I tend to agree, if the purpose is to test the controller and not the switches, this should be documented as part of the topology at the "start of the hour", so that the only equipment being tested is the controller.

>
>
>
>Third, we must consider the work currently being done by other 
>standards bodies. The Open Networking Foundation Testing & Interop 
>Working Group, for example, is currently developing it’s own OpenFlow 
>Controller benchmarks. I would prefer a combined solution  if possible.
>
>The draft under consideration should not go further until these issues 
>are addressed.

I partially agree, and I believe the authors might as well, in that we know other bodies are working on OpenFlow, for example. I do think some understanding of what's happening within those organizations would be an excellent starting point. However, I don't believe that this issue, with the 2 aforementioned issues, are a reason to stop proceeding with the document. If, through the process of working through this, we determine as a group, or the authors determine on their own, that what other groups are doing is sufficient, we can make that call at that time.  It's very early in the draft, and I wouldn't mind seeing where this takes us as a group, particularly if we can have some form of communication or understanding with other groups throughout the industry who might be looking at this problem in some way.

Kind regards,
Sarah