Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

"Castelli, Brian" <Brian.Castelli@spirent.com> Thu, 19 June 2014 04:18 UTC

Return-Path: <Brian.Castelli@spirent.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 85E5B1A01D6 for <bmwg@ietfa.amsl.com>; Wed, 18 Jun 2014 21:18:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.049
X-Spam-Level:
X-Spam-Status: No, score=0.049 tagged_above=-999 required=5 tests=[BAYES_50=0.8, HTML_MESSAGE=0.001, J_CHICKENPOX_32=0.6, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Eck1xyzuX6mc for <bmwg@ietfa.amsl.com>; Wed, 18 Jun 2014 21:18:25 -0700 (PDT)
Received: from webmail.spirent.com (smtp1.spirent.com [38.111.148.215]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9F1781A010D for <bmwg@ietf.org>; Wed, 18 Jun 2014 21:18:24 -0700 (PDT)
Received: from SPCCOREXCMBX01.AD.SPIRENTCOM.COM ([169.254.1.85]) by SPCCOREXCCAS02.AD.SPIRENTCOM.COM ([10.96.66.21]) with mapi id 14.02.0387.000; Wed, 18 Jun 2014 21:18:20 -0700
From: "Castelli, Brian" <Brian.Castelli@spirent.com>
To: "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance
Thread-Index: AQHPhQ0wPhnX4rvCTUmBdMpJ6z8vUg==
Date: Thu, 19 Jun 2014 04:18:19 +0000
Message-ID: <D21535D16E7F464EBA75B9CA12A7DFFD205C1B43@SPCCOREXCMBX01.AD.SPIRENTCOM.COM>
References: <CFB4B28A.541E%brian.castelli@spirent.com> <000001cf80d4$2fabbd00$8f033700$@veryxtech.com>
In-Reply-To: <000001cf80d4$2fabbd00$8f033700$@veryxtech.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.4.2.140509
x-originating-ip: [10.96.66.253]
Content-Type: multipart/related; boundary="_005_D21535D16E7F464EBA75B9CA12A7DFFD205C1B43SPCCOREXCMBX01A_"; type="multipart/alternative"
MIME-Version: 1.0
Archived-At: http://mailarchive.ietf.org/arch/msg/bmwg/XddySyWouUXwF9dGXwln5aDsbQ4
Cc: 'Anton Basil' <anton.basil@veryxtech.com>, 'Vishwas Manral' <vishwas.manral@gmail.com>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Jun 2014 04:18:30 -0000

I am sorry for the delay in responding. I have many additional comments on the draft, but I’d like to pop up a level before going into the specific details. In this message I will limit my comments to three high-level issues.

First, we must have an agreement on the terms and definitions used by a family of SDN benchmarking specifications. The definitions section of the current document has only two entries, yet there are terms used throughout the document that are defined in place or by context. As we produce a family of benchmarking specifications for SDN, all related documents must draw from a common set of definitions to avoid confusion and duplication. We should, therefore, produce a draft of definitions that can be referred to by this and subsequent benchmark specifications.

Second, we must ensure that the collection of tests in the draft are appropriate. Tests 6.1.1 and 6.1.2 are inverse of one another and therefore redundant. We should consolidate them into a single family of controller-response tests and simplify. Test 6.1.3 does not adequately isolate the controller in the test because the overall response time depends on the response time of the switches. If a tester runs the same test on the same controller with different switches, he or she will get different answers. The goal of a benchmark test is to enable apples-to-apples comparisons between controllers. I think a step back to reconsider the collection of proposed tests is in order.

Third, we must consider the work currently being done by other standards bodies. The Open Networking Foundation Testing & Interop Working Group, for example, is currently developing it’s own OpenFlow Controller benchmarks. I would prefer a combined solution if possible.

The draft under consideration should not go further until these issues are addressed.

From: "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com<mailto:bhuvaneswaran.vengainathan@veryxtech.com>>
Date: Thursday, June 5, 2014 at 11:38 AM
To: Brian Castelli <brian.castelli@spirent.com<mailto:brian.castelli@spirent.com>>, "bmwg@ietf.org<mailto:bmwg@ietf.org>" <bmwg@ietf.org<mailto:bmwg@ietf.org>>
Cc: 'Anton Basil' <anton.basil@veryxtech.com<mailto:anton.basil@veryxtech.com>>, 'Vishwas Manral' <vishwas.manral@gmail.com<mailto:vishwas.manral@gmail.com>>
Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Dear Brain,

Please find my responses inline to your comments and queries.

Thanks,

[Description: Description: Description: Description: Description: Description: Description: Description: Description: Veryx-Logo for Webinar]

Bhuvan | Cloud and SDN Solutions
Phone: +91 44 4567 2222 x242
Mobile: +91 988 428 6693
www.veryxtech.com<http://www.veryxtech.com/>

From: Castelli, Brian [mailto:Brian.Castelli@spirent.com]
Sent: Wednesday, June 04, 2014 8:59 PM
To: Castelli, Brian; Bhuvan (Veryx Technologies); bmwg@ietf.org<mailto:bmwg@ietf.org>
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

I forgot one comment…

Section 6.1.1.1.5 reporting format:

The last sentence of the first paragraph says, “The test report MUST indicate whether learning was enabled or disabled.” What type of learning does this requirement refer to? And, if it’s important, shouldn’t it be a Setup Parameter listed in section 6.1.1.1.2?
[Bhuvan] Learning here indicates MAC/IP address learning. As suggested, this can be made as setup parameters.

From: <Castelli>, Brian Castelli <brian.castelli@spirent.com<mailto:brian.castelli@spirent.com>>
Date: Wednesday, June 4, 2014 at 11:15 AM
To: "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com<mailto:bhuvaneswaran.vengainathan@veryxtech.com>>, "bmwg@ietf.org<mailto:bmwg@ietf.org>" <bmwg@ietf.org<mailto:bmwg@ietf.org>>
Cc: 'Anton Basil' <anton.basil@veryxtech.com<mailto:anton.basil@veryxtech.com>>, 'Vishwas Manral' <vishwas.manral@gmail.com<mailto:vishwas.manral@gmail.com>>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Bhuvan,

Thank you for the response and the advice on how to best participate in the discussion. I have a few follow-up comments/questions:
Section 6.1.1.1.3 Procedure:
How is it that we are going to assist the person running the test at creating the proper packet-in messages? What are the characteristics of the packet-ins that will make this test successful?
[Bhuvan] To make the test successful, we need to simulate packet-in messages for each flow. Typically flows are identified using dst.mac and dst.ip.
[Brian] Is this well known to test authors? Should the document provide guidance about how to create packet_in messages that correspond to the desired flows?

[Bhuvan] We can provide an example on how to create packet_in messages for a desired flows.
 In the reactive mode setup, there is no mention of the rate at which packet-ins are sent, but the test result can be skewed by the rate. If I send only one packet-in per second, for example, the FSR can’t be faster than one flow per second. I think we should consider specifying the rate at which the packet-ins are sent to the controller, iterating over packet-in rates, iterating over numbers of switches.
[Bhuvan] We agree with your comment. We are planning to define the minimum number of packet-ins that need to be sent on each OpenFlow connection (typically greater than 1). But the assumption is that we need to send as many packet-in messages as possible on an established OpenFlow connection to measure the FSR.
 [Brian] I think it would be interesting to iterate over different packet_in rates. I would expect controller response time to degrade with increased packet_in load. An iteration would allow us to determine how well the controller scales and how gracefully response time degrades.
[Bhuvan] We agree with your feedback. We have covered this scenario as part of section 6.1.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-00#section-6.1.1.2>  - Flow setup rate under incremental network load.

  *   Without the iteration I have suggested, above, I don’t understand why the test should be repeated multiple times? Do we not expect the test results to be consistent between test runs?
[Bhuvan] We expect slight variations across test runs. Hence we are repeating the test multiple times to provide tester a reliable metric.
[Brian] Personal experience says that testers do not care about slight variations. They want to run the test and get an answer, not run the test 10 times and find that variation is less than 1% across all tests. It is far more interesting for a tester when there is a variable that changes between iterations, as I have suggested, above. Running the same exact test multiple times is usually a waste of test resources.
[Bhuvan] For testing controller functionality, the obtained results would be consistent across iterations. But for the performance tests, the behavior of system would not be consistent and we recommend iterating the tests for atleast 3 times to provide tester a reliable metric.
Section 6.1.1.1.4 Measurement:

  *   FSRi equation is incorrect. FSR = number of flows/time, not OFR/time.
[Bhuvan] I would like to provide some more clarification on this equation. Here, the number of flows = number of packet-in messages sent to the controller. We have conducted this test with various controllers and observed that the number of OF responses received from the controller is always lesser than the number of packet-in messages sent to the controller due to the TCP window restrictions. Hence we recommend, FSR should be measured based on the OF responses received from the controller. The FSR equation will therefore be OF responses/time.
 [Brian] This is my mistake. I jumped to the conclusion that ‘R’ in OFR was rate. Looking back at section 2, Terminology, OFR = OpenFlow responses. I thought the equation was rate/time. My bad. I withdraw the comment.

E-mail confidentiality.
--------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US,

Spirent Communications,
26750 Agoura Road, Calabasas, CA, 91302, USA.
Tel No. 1-818-676- 2300