Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance
"Bhuvan \(Veryx Technologies\)" <bhuvaneswaran.vengainathan@veryxtech.com> Wed, 04 June 2014 14:57 UTC
Return-Path: <bhuvaneswaran.vengainathan@veryxtech.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0CCD81A0290 for <bmwg@ietfa.amsl.com>; Wed, 4 Jun 2014 07:57:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.443
X-Spam-Level:
X-Spam-Status: No, score=0.443 tagged_above=-999 required=5 tests=[BAYES_05=-0.5, HTML_MESSAGE=0.001, J_CHICKENPOX_32=0.6, RELAY_IS_203=0.994, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jAn2GLcWZSuf for <bmwg@ietfa.amsl.com>; Wed, 4 Jun 2014 07:57:02 -0700 (PDT)
Received: from mail.veryxtech.com (mail.veryxtech.com [203.196.171.45]) by ietfa.amsl.com (Postfix) with ESMTP id EA1901A0080 for <bmwg@ietf.org>; Wed, 4 Jun 2014 07:57:00 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by mail.veryxtech.com (Postfix) with ESMTP id 5263F3740DE; Wed, 4 Jun 2014 20:26:53 +0530 (IST)
X-Virus-Scanned: amavisd-new at veryxtech.com
Received: from mail.veryxtech.com ([127.0.0.1]) by localhost (mail.veryxtech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id q6UO5Fwg15NC; Wed, 4 Jun 2014 20:26:41 +0530 (IST)
Received: from LT015PC (unknown [192.168.12.87]) by mail.veryxtech.com (Postfix) with ESMTPSA id 9F73E3740D1; Wed, 4 Jun 2014 20:26:41 +0530 (IST)
From: "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com>
To: Brian.Castelli@spirent.com, bmwg@ietf.org
References: <CFB2B4B8.52D9%brian.castelli@spirent.com>
In-Reply-To: <CFB2B4B8.52D9%brian.castelli@spirent.com>
Date: Wed, 04 Jun 2014 20:26:41 +0530
Message-ID: <003c01cf8005$31d099e0$9571cda0$@veryxtech.com>
MIME-Version: 1.0
Content-Type: multipart/related; boundary="----=_NextPart_000_003D_01CF8033.4B924BC0"
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQIj3j5OTwbcoRHRvVEERF4r2Aso7Jq4AHHA
Content-Language: en-us
Archived-At: http://mailarchive.ietf.org/arch/msg/bmwg/HOTxu7hPOg6olAqmb2oMoUzuX_E
Cc: 'Anton Basil' <anton.basil@veryxtech.com>, 'Vishwas Manral' <vishwas.manral@gmail.com>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Jun 2014 14:57:06 -0000
Dear Brain, Thank you for your valuable comments. Please see my responses inline to your comments. Trust my responses address your review comments. Please revert back if you need any clarifications on my responses. Look forward your additional comments and suggestions on this draft. Best Regards, Description: Description: Description: Description: Description: Description: Description: Description: Description: Veryx-Logo for Webinar Bhuvan | Cloud and SDN Solutions Phone: +91 44 4567 2222 x242 Mobile: +91 988 428 6693 <http://www.veryxtech.com/> www.veryxtech.com From: Castelli, Brian [mailto:Brian.Castelli@spirent.com] Sent: Wednesday, June 04, 2014 3:30 AM To: anton.basil@veryxtech.com; bhuvaneswaran.vengainathan@veryxtech.com; vishwas.manral@gmail.com Subject: Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance Dear Authors, I have just joined the mailing list today. I don't know how to respond back to the mail thread after the fact, so I am sending this feedback on the SDN Controller benchmark document directly to you. [Bhuvan] I'm including IETF mail id in the CC of my response. In future you can share your comments to the email id - bmwg@ietf.org. I read the document with great interest because I am working on SDN Controller benchmarks myself. I have many comments and suggestions for improving the document. Because I am new to the IETF process, I am sending you comments through page 10 to validate that these are the kinds of comments you are soliciting. I don't want to waste your time with comments that are not constructive. [Bhuvan] We will be happy to receive your comments and suggestions for improving the draft. Here are my comments on the document pages 1-10: 5.3. Southbound Connection Setup Requirements: Seems awkward to refer to "channel setup ways". I prefer "channel configuration". The first sentence would then read as follows: "The methodologies defined in this document are independent of OpenFlow version, channel configuration, and connection type (e.g. TCP or TLS/SSL)." Note the suggested punctuation changes as well. [Bhuvan]We accept your suggestion. We are planning to rephrase the sentence as "The methodologies defined in this document are independent of OpenFlow version, channel setup configuration (Single/Multiple/Auxillary connections), and connection type (e.g. TCP or TLS/SSL)." 6.1.1 Flow setup rate: For easier reading in subsequent paragraphs, I suggest that you add the acronym to the title, as in "Flow Setup Rate (FSR)". [Bhuvan] We will add the acronym as per your suggestion. 6.1.1 Flow setup rate: I think the paragraph after the title should not specify "maximum". Testing could be for max, average, minimum, etc. So the text ought to read as, "The FSR is the number of flows transmitted by a controller, expressed in flows per second." [Bhuvan] We will incorporate this feedback in the draft. 6.1.1.1 Flow setup rate under constant network load: Use the acronym proposed above. Should say, "FSR under constant network load" [Bhuvan] As suggested, we will change the title. 6.1.1.1.1 Objective: The test does not measure the total number. It measures the rate. Plus the test only measures responses in the reactive mode case, not in the proactive mode case. It would be more correct to write, "To measure the rate of OpenFlow flow setup messages transmitted by a controller under constant network load." [Bhuvan] This is a valid feedback. We will rephrase as "To measure the total number of OpenFlow flow responses (Packet-Out/Flow-Mod) received per second from the controller under constant network load. 6.1.1.1.2 Setup Parameters: * Number of flows is an OpenFlow setup parameter, not a network setup parameter [Bhuvan] Number of flows here defines the number of network flows that the controller can setup on OpenFlow connection. We will therefore rephrase as "Defines unique number of network flows setup on each OpenFlow connections" * Test Iterations: "need" should be "needs" * Test Duration: "duration of test iteration" should be "duration of each test iteration" [Bhuvan] We will change as per your suggestion. Section 6.1.1.1.3 Procedure: * In a few places we have written, "all the specified number of switches," when, "the specified number of switches" would be more correct. [Bhuvan] We will change as per your suggestion. * In the reactive mode setup, the text reads, "Load the controller with Packet-In messages send from all switches for a configured number of flows and for a specified test duration." "send" should be "sent". [Bhuvan] We will make the suggested correction. And what does it mean to send packet-in messages "for a configured number of flows"? Do you mean to say to send packet-ins that correspond to a number of desired flows? [Bhuvan] Yes Do we need to configure the controller to send down a new flow for every new destination IP address received or something? [Bhuvan] No need to configure the controller. Controllers by default respond to packet-in messages with packet-out/flow-mod messages. How is it that we are going to assist the person running the test at creating the proper packet-in messages? What are the characteristics of the packet-ins that will make this test successful? [Bhuvan] To make the test successful, we need to simulate packet-in messages for each flow. Typically flows are identified using dst.mac and dst.ip. * In the reactive mode setup, there is no mention of the rate at which packet-ins are sent, but the test result can be skewed by the rate. If I send only one packet-in per second, for example, the FSR can't be faster than one flow per second. I think we should consider specifying the rate at which the packet-ins are sent to the controller, iterating over packet-in rates, iterating over numbers of switches. [Bhuvan] We agree with your comment. We are planning to define the minimum number of packet-ins that need to be sent on each OpenFlow connection (typically greater than 1). But the assumption is that we need to send as many packet-in messages as possible on an established OpenFlow connection to measure the FSR. * For the proactive mode, why is there a test duration? We ought to configure the controller to push down a number of flows, connect the proper number of switches, and measure how long it takes to send all the flows without error. [Bhuvan] In proactive mode, we typically measure how much flows a controller can provision per second. Hence we need the test duration. * Without the iteration I have suggested, above, I don't understand why the test should be repeated multiple times? Do we not expect the test results to be consistent between test runs? [Bhuvan] We expect slight variations across test runs. Hence we are repeating the test multiple times to provide tester a reliable metric. Section 6.1.1.1.4 Measurement: * FSRi equation is incorrect. FSR = number of flows/time, not OFR/time. [Bhuvan] I would like to provide some more clarification on this equation. Here, the number of flows = number of packet-in messages sent to the controller. We have conducted this test with various controllers and observed that the number of OF responses received from the controller is always lesser than the number of packet-in messages sent to the controller due to the TCP window restrictions. Hence we recommend, FSR should be measured based on the OF responses received from the controller. The FSR equation will therefore be OF responses/time. Section 6.1.1.1.5 Reporting: * Without iteration over the variables I have suggested, above, running the same exact test multiple times would yield a flat line. I don't understand why we would graph that kind of result. We should use a graph when there are variables. [Bhuvan] As explained above, we recommend iterating the test for multiple times as we see variations in the results. I will withhold additional comments until I hear back from you. Thank you, Brian E-mail confidentiality. -------------------------------- This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system. Spirent Communications plc Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom. Tel No. +44 (0) 1293 767676 Fax No. +44 (0) 1293 767677 Registered in England Number 470893 Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom. Or if within the US, Spirent Communications, 26750 Agoura Road, Calabasas, CA, 91302, USA. Tel No. 1-818-676- 2300
- Re: [bmwg] Feedback for Benchmarking Methodology … Bhuvan (Veryx Technologies)
- Re: [bmwg] Feedback for Benchmarking Methodology … Castelli, Brian
- Re: [bmwg] Feedback for Benchmarking Methodology … Castelli, Brian
- Re: [bmwg] Feedback for Benchmarking Methodology … Bhuvan (Veryx Technologies)
- Re: [bmwg] Feedback for Benchmarking Methodology … Castelli, Brian
- Re: [bmwg] Feedback for Benchmarking Methodology … Banks, Sarah
- Re: [bmwg] Feedback for Benchmarking Methodology … Bhuvan (Veryx Technologies)
- Re: [bmwg] Feedback for Benchmarking Methodology … Castelli, Brian
- Re: [bmwg] Feedback for Benchmarking Methodology … MORTON, ALFRED C (AL)
- Re: [bmwg] Feedback for Benchmarking Methodology … Bhuvan (Veryx Technologies)
- Re: [bmwg] Feedback for Benchmarking Methodology … Castelli, Brian
- Re: [bmwg] Feedback for Benchmarking Methodology … Bhuvan (Veryx Technologies)