Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

["Bhuvan \(Veryx Technologies\)" <[e-mail]>]

Dear Brain,

 

Please find my responses inline to your comments and queries.

 

Thanks,

 

Description: Description: Description: Description: Description:
Description: Description: Description: Description: Veryx-Logo for Webinar



Bhuvan | Cloud and SDN Solutions

Phone: [phone] x242

Mobile: [phone] 

 <http://www.veryxtech.com/> www.veryxtech.com

 

From: Castelli, Brian [mailto:[e-mail]] 
Sent: Wednesday, June 04, 2014 8:59 PM
To: Castelli, Brian; Bhuvan (Veryx Technologies); [e-mail]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

 

I forgot one comment.

 

Section 6.1.1.1.5 reporting format:

 

The last sentence of the first paragraph says, "The test report MUST
indicate whether learning was enabled or disabled." What type of learning
does this requirement refer to? And, if it's important, shouldn't it be a
Setup Parameter listed in section 6.1.1.1.2?

[Bhuvan] Learning here indicates MAC/IP address learning. As suggested, this
can be made as setup parameters.

 

From: <Castelli>, Brian Castelli <[e-mail]>
Date: Wednesday, June 4, 2014 at 11:15 AM
To: "Bhuvan (Veryx Technologies)"
<[e-mail]>, "[e-mail]" <[e-mail]>
Cc: 'Anton Basil' <[e-mail]>, 'Vishwas Manral'
<[e-mail]>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

 

Bhuvan,

 

Thank you for the response and the advice on how to best participate in the
discussion. I have a few follow-up comments/questions:

Section 6.1.1.1.3 Procedure: 

How is it that we are going to assist the person running the test at
creating the proper packet-in messages? What are the characteristics of the
packet-ins that will make this test successful?

[Bhuvan] To make the test successful, we need to simulate packet-in messages
for each flow. Typically flows are identified using dst.mac and dst.ip.

[Brian] Is this well known to test authors? Should the document provide
guidance about how to create packet_in messages that correspond to the
desired flows?

 

[Bhuvan] We can provide an example on how to create packet_in messages for a
desired flows.

 In the reactive mode setup, there is no mention of the rate at which
packet-ins are sent, but the test result can be skewed by the rate. If I
send only one packet-in per second, for example, the FSR can't be faster
than one flow per second. I think we should consider specifying the rate at
which the packet-ins are sent to the controller, iterating over packet-in
rates, iterating over numbers of switches.

[Bhuvan] We agree with your comment. We are planning to define the minimum
number of packet-ins that need to be sent on each OpenFlow connection
(typically greater than 1). But the assumption is that we need to send as
many packet-in messages as possible on an established OpenFlow connection to
measure the FSR.

 [Brian] I think it would be interesting to iterate over different packet_in
rates. I would expect controller response time to degrade with increased
packet_in load. An iteration would allow us to determine how well the
controller scales and how gracefully response time degrades.

[Bhuvan] We agree with your feedback. We have covered this scenario as part
of section
<http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-00#
section-6.1.1.2> 6.1.1.2  - Flow setup rate under incremental network load.

*	Without the iteration I have suggested, above, I don't understand
why the test should be repeated multiple times? Do we not expect the test
results to be consistent between test runs?

[Bhuvan] We expect slight variations across test runs. Hence we are
repeating the test multiple times to provide tester a reliable metric.

[Brian] Personal experience says that testers do not care about slight
variations. They want to run the test and get an answer, not run the test 10
times and find that variation is less than 1% across all tests. It is far
more interesting for a tester when there is a variable that changes between
iterations, as I have suggested, above. Running the same exact test multiple
times is usually a waste of test resources.

[Bhuvan] For testing controller functionality, the obtained results would be
consistent across iterations. But for the performance tests, the behavior of
system would not be consistent and we recommend iterating the tests for
atleast 3 times to provide tester a reliable metric.

Section 6.1.1.1.4 Measurement:

*	FSRi equation is incorrect. FSR = number of flows/time, not
OFR/time.

[Bhuvan] I would like to provide some more clarification on this equation.
Here, the number of flows = number of packet-in messages sent to the
controller. We have conducted this test with various controllers and
observed that the number of OF responses received from the controller is
always lesser than the number of packet-in messages sent to the controller
due to the TCP window restrictions. Hence we recommend, FSR should be
measured based on the OF responses received from the controller. The FSR
equation will therefore be OF responses/time.

 [Brian] This is my mistake. I jumped to the conclusion that 'R' in OFR was
rate. Looking back at section 2, Terminology, OFR = OpenFlow responses. I
thought the equation was rate/time. My bad. I withdraw the comment.

 

E-mail confidentiality.
--------------------------------
This e-mail contains confidential and / or privileged information belonging
to Spirent Communications plc, its affiliates and / or subsidiaries. If you
are not the intended recipient, you are hereby notified that any disclosure,
copying, distribution and / or the taking of any action based upon reliance
on the contents of this transmission is strictly forbidden. If you have
received this message in error please notify the sender by return e-mail and
delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United
Kingdom.
Tel No. [phone]
Fax No. [phone]

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN,
United Kingdom.

Or if within the US, 

Spirent Communications,
26750 Agoura Road, Calabasas, CA, 91302, USA.
Tel No. 1-818-676- 2300