Re: [bmwg] Benchmarking Methodology for OpenFlow SDN Controller Performance

["Bhuvan \(Veryx Technologies\)" <[e-mail]>]

Hi Mark,

 

Thanks for reviewing the draft (initial sections) and providing valuable
comments.

Please find my responses inline with tag [Bhuvan].

If needed, we can discuss my responses.

 

Also, I look forward your comments on other sections of the draft (if any).

 

Best regards,

 

Description: Description: Description: Description: Description:
Description: Description: Description: Description: Veryx-Logo for Webinar



Bhuvan | Cloud and SDN Solutions

Phone: [phone] x242

Mobile: [phone] 

 <http://www.veryxtech.com/> www.veryxtech.com

 

Description: Description: Description: Description: Description:
Description: Description: Description: Signature

 

From: bmwg [mailto:[e-mail]] On Behalf Of Tassinari, Mark A
Sent: Saturday, April 26, 2014 4:39 AM
To: [e-mail]
Subject: [bmwg] Benchmarking Methodology for OpenFlow SDN Controller
Performance

 

Hello Bhuvan, 

 

Standardizing on a set of controller performance tests is a great idea and
the OpenFlow SDN Controller Performance draft is a good start.  I'd like to
submit some comments and questions.  

 

-          The Test Setup section is unclear about where virtualization is
permitted but this can have a dramatic effect on the results.  I believe the
benchmark test should require the controller and any virtual switches to be
on separate servers.  

[Bhuvan] I do agree with your comment. The controller and the emulated
virtual switches need to be running in separate servers. We will explicitly
capture this point as part of test setup in our next version.

 

-          I'm curious to know why the number of switches in the tests are
not defined - not even to require more than one.  There is way to know if
the tests were executed the tests under the same conditions which makes
comparison much less meaningful.

[Bhuvan] I agree with you that the number switches should be more than one
(suggestion could be - max. switches supported by the controller).  We can
add a note to specify this in the draft.
But the reasons behind not defining the exact value for number of switches
are:
1) The number of switches supported may vary from controller to controller
2) Some controllers may perform better for smaller number of switches and
some for larger number of switches
Hence we left it to the vendors to specify this parameter based on their
implementation. Also we recommended specifying the results against the
number of switches emulated as part of "Reporting Format" section to enable
comparison.

 

-          What is the numerator (OFRi) in the Flow setup rate formula
defined in Section 6.1.1.1.4?  Is it the sum of all packet_out and flow mod
messages sent by the controller? 

[Bhuvan] No. We recommend taking into account either sum of all 'PACKET_OUT'
(or) 'FLOW_MOD' messages. We can also provide description for OFRi in
Terminology section. Please let me know if you have different thoughts on
this. We could discuss them.

 

-          The test does not specify whether throughput results are verified
to be lossless (every packet_in results in a packet_out).  Since TCP is the
specified connection it may not matter but I thought I'd toss it out for
discussion.  

[Bhuvan] The Throughput results are verified based on the total number of
responses received from the controller over the sample interval (not
considering every packet_ins results in a packet_out). We may look at adding
this as an additional metric if the end users are looking for such metrics.
As you pointed out, we shall leave this open for discussion.

 

-          The Reactive Mode Flow setup in section 6.1.1.2.2 is a little
unclear.  If the objective is to have the 10% of all "specified" switches
that are connected send Packet_In messages I'd rephrase  the second sentence
to state "Load controller with Packet-In messages from all connected
controllers for a specified interval."  

[Bhuvan] As suggested, we will rephrase in next version of draft.

 

Regards, 

 

Mark