Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

"Bhuvan \(Veryx Technologies\)" <bhuvaneswaran.vengainathan@veryxtech.com> Mon, 23 June 2014 14:53 UTC

Return-Path: <bhuvaneswaran.vengainathan@veryxtech.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B058D1B296A for <bmwg@ietfa.amsl.com>; Mon, 23 Jun 2014 07:53:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.142
X-Spam-Level: *
X-Spam-Status: No, score=1.142 tagged_above=-999 required=5 tests=[BAYES_50=0.8, RELAY_IS_203=0.994, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oxC08hXPNsKq for <bmwg@ietfa.amsl.com>; Mon, 23 Jun 2014 07:53:16 -0700 (PDT)
Received: from mail.veryxtech.com (mail.veryxtech.com [203.196.171.45]) by ietfa.amsl.com (Postfix) with ESMTP id 5DF621B2965 for <bmwg@ietf.org>; Mon, 23 Jun 2014 07:53:16 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by mail.veryxtech.com (Postfix) with ESMTP id 9A91F374110; Mon, 23 Jun 2014 20:23:14 +0530 (IST)
X-Virus-Scanned: amavisd-new at veryxtech.com
Received: from mail.veryxtech.com ([127.0.0.1]) by localhost (mail.veryxtech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qaR+s36rHJsF; Mon, 23 Jun 2014 20:23:09 +0530 (IST)
Received: from LT015PC (unknown [192.168.12.102]) by mail.veryxtech.com (Postfix) with ESMTPSA id A28563740DF; Mon, 23 Jun 2014 20:23:09 +0530 (IST)
From: "Bhuvan (Veryx Technologies)" <bhuvaneswaran.vengainathan@veryxtech.com>
To: "'MORTON, ALFRED C (AL)'" <acmorton@att.com>, "'Castelli, Brian'" <Brian.Castelli@spirent.com>, "'Banks, Sarah'" <sbanks@akamai.com>, bmwg@ietf.org
References: <CFB4B28A.541E%brian.castelli@spirent.com> <000001cf80d4$2fabbd00$8f033700$@veryxtech.com> <D21535D16E7F464EBA75B9CA12A7DFFD205C1B43@SPCCOREXCMBX01.AD.SPIRENTCOM.COM> <CFC82A01.5BC1%sbanks@akamai.com> <CFC86AB3.6288%brian.castelli@spirent.com> <2845723087023D4CB5114223779FA9C801896A83AD@njfpsrvexg8.research.att.com>
In-Reply-To: <2845723087023D4CB5114223779FA9C801896A83AD@njfpsrvexg8.research.att.com>
Date: Mon, 23 Jun 2014 20:23:18 +0530
Message-ID: <007301cf8ef2$dfc6eee0$9f54cca0$@veryxtech.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQLIjOLu12eMUttGsbxYXm1U/hz76gIAKjECAWjlDfwBKvg6DwMs9I6fAclgs9OZQHfUYA==
Content-Language: en-us
Archived-At: http://mailarchive.ietf.org/arch/msg/bmwg/rrLXAVK_rB-Juz12UHmDmJv4JIM
Cc: 'Anton Basil' <anton.basil@veryxtech.com>, 'Vishwas Manral' <vishwas.manral@gmail.com>
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Jun 2014 14:53:20 -0000

Hi Al,

Thanks for your comment. I do agree with you that there needs to be a common
benchmarking mechanism for controller designs performing the same tasks.
But the technology is still relatively immature and the approach has various
dimensions including centralized control vs distributed control (controller
less).
Again the centralized control approach uses different programming methods
(OpenFlow vs Non OpenFlow).

Having said that, for better usability/understanding of the draft and to
avoid misinterpretation, I personally feel it would be good to have separate
benchmarking methodology for each approach (though the metrics remain same).
Also we would be happy to continue the effort to extend the same metrics for
other approaches too.

Please let me know if I've missed something.

Sarah/Brain, please also share your thoughts on this.

Thanks,
Bhuvan

-----Original Message-----
From: MORTON, ALFRED C (AL) [mailto:acmorton@att.com] 
Sent: Saturday, June 21, 2014 6:00 PM
To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx Technologies);
bmwg@ietf.org
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

Hi Brian, Sarah, Bhuvan, and all,

a couple of quick answers and a comment:

> -----Original Message-----
> From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Castelli, Brian
...
> My two main concerns are getting the terminology right and making sure 
> that the proposed set of tests will achieve our goals. I believe those 
> goals ought to include a common taxonomy, enabling apples-to-apples 
> comparisons, and minimizing the time/work required to execute the test 
> cases.

Great, that's exactly what we are chartered to do in BMWG.
http://datatracker.ietf.org/wg/bmwg/charter/


> 
> I am willing to help improve the benchmark. I understand that it is an 
> early draft. The time is right to make sure that we are developing the 
> best specification that we can.
> 
> How do we continue this process? Via email to this list? I am willing 
> to help.

It's principally e-mail (just as you've been doing) and face-to-face
meetings three times a year.  Sarah mentions that our next meeting is in her
home town:
http://www.ietf.org/meeting/90/index.html

Make some text proposals for the draft, and we can discuss them on the list.

Having said that, and recognizing that there appears to be a comparable
activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to add
value to the industry is to approach the SDN controller problem more
generically, such that different controller designs performing the same
tasks could be benchmarked as black-boxes.  I realize this approach has
substantial implications for the draft, but the benefit of wider
applicability.

food for thought,
Al
(as a participant)