Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version
"Bhuvan \(Veryx Technologies\)" <bhuvaneswaran.vengainathan@veryxtech.com> Fri, 07 August 2015 11:32 UTC
Return-Path: <bhuvaneswaran.vengainathan@veryxtech.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 521391B2AB4 for <bmwg@ietfa.amsl.com>; Fri, 7 Aug 2015 04:32:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.916
X-Spam-Level:
X-Spam-Status: No, score=-0.916 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RELAY_IS_203=0.994, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FbI86_Va57_J for <bmwg@ietfa.amsl.com>; Fri, 7 Aug 2015 04:32:08 -0700 (PDT)
Received: from mail.veryxtech.com (mail.veryxtech.com [203.196.171.45]) by ietfa.amsl.com (Postfix) with ESMTP id 670F41B2AAD for <bmwg@ietf.org>; Fri, 7 Aug 2015 04:32:07 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by mail.veryxtech.com (Postfix) with ESMTP id 47DAF3740DF; Fri, 7 Aug 2015 17:02:05 +0530 (IST)
X-Virus-Scanned: amavisd-new at veryxtech.com
Received: from mail.veryxtech.com ([127.0.0.1]) by localhost (mail.veryxtech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7eRHJxsD9N9a; Fri, 7 Aug 2015 17:01:53 +0530 (IST)
Received: from LT015 (unknown [192.168.12.89]) by mail.veryxtech.com (Postfix) with ESMTPSA id 365FB3740D1; Fri, 7 Aug 2015 17:01:53 +0530 (IST)
From: "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com>
To: 'Khasanov Boris' <khasanov.boris@huawei.com>
References: <007301d0c3a7$9a13fba0$ce3bf2e0$@veryxtech.com> <000001d0cc32$7c8f1f90$75ad5eb0$@is.naist.jp> <8703B772-E5CA-4933-8799-8B653FBBB31E@encrypted.net> <C7794D4A32C7D046B93DBCF0FA202C18FC2211@lhreml503-mbx>
In-Reply-To: <C7794D4A32C7D046B93DBCF0FA202C18FC2211@lhreml503-mbx>
Date: Fri, 07 Aug 2015 17:01:52 +0530
Message-ID: <004101d0d104$a8ffb8b0$faff2a10$@veryxtech.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_NextPart_000_0042_01D0D132.C2C429B0"
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQJu+aLp35MXVPA9pBpjm1ci9f/zyAHmDns2AW5isNwCTqtS5pyW7oeQ
Content-Language: en-in
Archived-At: <http://mailarchive.ietf.org/arch/msg/bmwg/CuU1cT0EtPZU7gV-kTK5cbUuUCY>
Cc: bmwg@ietf.org
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Aug 2015 11:32:13 -0000
Hi Boris, Thank you very much for taking time to review this draft and providing your valuable comments. I've provided my responses in line to your comments. Best regards, Bhuvan From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Khasanov Boris Sent: 07 August 2015 12:37 To: Sarah Banks Cc: bmwg@ietf.org Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version Hi Sarah and all, I am sorry for jumping late into discussion. Just few thoughts or my 0,02$: 1) <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 01#section-4.2> 4.2 Test Traffic. IMHO, it would be better to say there explicitly that in case of performance testing of forwarding plane we are looking for NDRs for different frame sizes according to RFC2544, if I got it correct. [Bhuvan] I would like to clarify that the draft focusses on defining test methodology for control plane. So we believe that measuring NDRs for different frame sizes would not be applicable. But it is important to perform the defined tests for various frame sizes and traffic type. So which we are referring to RFC 2544. 2) <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 01#section-6.1.2> 6.1.2 Asynchronous Message Processing Time May be would be good to add a couple of words for clarification - why it is important to SDN controller to process such messages as Error or Flow-Removed as quick as possible. It may sound obvious, but I believe that for many customers such clarification will be important. [Bhuvan] Sure. We will add some clarification in the discussion section (2.3.1.2) of draft-bhuvan-bmwg-sdn-controller-benchmark-term-00 draft. 3) <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 01#section-6.1.2> 6.1.2 Asynchronous Message Processing Time Would it make sense to specify explicitly which messages need to be generated? I am not SW engineer so please correct me if my understanding is wrong, but I think different async messages may have different processing time on SDN controller, so results may be different and hard to compare (like comparison of best results but these results will be collected for different messages on different controllers). We also have 6.3.1 test which measures correlation between error packets and performance which provides more details about messages vs. 6.1.2 4) Same comment as previous one for 6.1.3 [Bhuvan] I completely agree with you. But the draft defines methodology generic to all controllers implementation, I'm not sure if we able to explicitly specify/recommend any particular message. We will provide some examples for reference. 5) 6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test procedure says about sending single traffic stream between end nodes. But it does not say about traffic stream parameters (frame size, rate, IPv4 or something else etc.). Would be good to specify it IMHO (as I mentioned above in 4.2 are not many details in current draft) . Another option could be just comment that these parameters should be defined before testing by customer itself. [Bhuvan] Typically controller provisions the network path for a flow in the SDN nodes (switches) when it sees the first packet of a flow (reactive). Once the path is provisioned, all the subsequent packets are forwarded by the SDN node (e.g., switch) based on the provisioned flow. Here we are measuring the time that the controller takes to pave the entire path for a flow in the SDN nodes. So I believe that the packet size or the rate may not influence the test. 6) 6.1.6 and 6.1.7 Reactive and Proactive rates. Same question as in 5. Also here we have to send traffic with some rate and calculate rate at the receiver end. Sounds that we are looking for NDR here, may be we need to mention it? Because in other tests like 6.4.1 (Procedure Step1) we actually say that traffic should be sent with NDR, so we need to find it during baseline tests. [Bhuvan] Sure. We will mention the rate. 7) 6.1.8 Network Topology Change Detection Time. Procedure Step 1, should we bring down ANY active node? Does not matter which one? [Bhuvan] This will not matter for this test. 8) 6.2 Scalability tests. Shouldn't we put there multidimensional test to find out the maximum numbers of parameters from 6.2.1-6.2.3 but altogether? [Bhuvan] Could you please provide some more clarification about this comment? 9) 6.4.2 Network Re-Provisioning Time. Again it says about sending bi-directional traffic stream but without any stream details. [Bhuvan] We will provide the details. Thank you. SY, Boris From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Sarah Banks Sent: Sunday, August 02, 2015 7:08 AM To: Marius Georgescu Cc: bmwg@ietf.org Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version Thank you Marius, for your time and inline suggestions. We'll review and touch base soon. Thanks Sarah On Aug 1, 2015, at 1:17 AM, Marius Georgescu <liviumarius-g@is.naist.jp> wrote: Dear Bhuvan et alia, Please find attached my inline suggestions for the new draft. They were marked with ###MG . Thank you for taking care of so many comments. Best regards, Marius From: bmwg [ <mailto:bmwg-bounces@ietf.org> mailto:bmwg-bounces@ietf.org] On Behalf Of Bhuvan (Veryx Technologies) Sent: Tuesday, July 21, 2015 8:23 PM To: <mailto:bmwg@ietf.org> bmwg@ietf.org Cc: 'Anton Basil' < <mailto:anton.basil@veryxtech.com> anton.basil@veryxtech.com>; Tassinari, Mark A < <mailto:mark.tassinari@hp.com> mark.tassinari@hp.com>; vishwas.manral < <mailto:vishwas.manral@gmail.com> vishwas.manral@gmail.com> Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version Dear BMWG Members, We have updated the draft ( <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 00> draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00) about SDN Controller benchmarking addressing comments received in IETF-92 meeting. Thank you very much for providing your valuable comments. The latest draft can be found in <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 01> draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 Summary of Changes: a. Updated test setup diagram following the comment from Scott Bradner. b. Added recommendations for test topology, test iterations etc., to use for benchmarking. c. Provided reference test topologies. d. Split Path Provisioning tests into two different tests - Proactive and Reactive Path Provisioning tests. e. Provided more clarity on test procedure for some of the tests. f. Fixed IETF normative language usage. We would love to hear any comments and queries on the same. Thanks, Authors <draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 MG.TXT>_______________________________________________ bmwg mailing list <mailto:bmwg@ietf.org> bmwg@ietf.org <https://www.ietf.org/mailman/listinfo/bmwg> https://www.ietf.org/mailman/listinfo/bmwg
- [bmwg] Benchmarking Methodology for SDN Controlle… Bhuvan (Veryx Technologies)
- [bmwg] Benchmarking Methodology for SDN Controlle… Bhuvan (Veryx Technologies)
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Marius Georgescu
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Sarah Banks
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Khasanov Boris
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Bhuvan (Veryx Technologies)
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Khasanov Boris
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Bhuvan (Veryx Technologies)
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Tassinari, Mark
- Re: [bmwg] Benchmarking Methodology for SDN Contr… Khasanov Boris