Re: [bmwg] Benchmarking Methodology for SDN Controller Performance

Hi Sarah and all,
I am sorry  for jumping late into discussion.
Just few thoughts or my 0,02$:

1)      4.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-4.2> Test Traffic. IMHO, it would be better to say there explicitly  that in case of performance testing of forwarding plane we are looking for NDRs for different frame sizes according to RFC2544, if I got it correct.

2)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time May be would be good to add a couple of words for clarification - why it is important to SDN controller to process such messages as Error or Flow-Removed as quick as possible. It may sound obvious, but I believe that for many customers such clarification will be important.

3)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time Would it make sense to specify explicitly which messages need to be generated? I am not SW engineer so please correct me if my understanding is wrong, but I think different async messages may have different processing time on SDN controller, so results may be different and hard to compare (like comparison of best results but these results will be collected for different messages on different controllers).  We also have 6.3.1 test which measures correlation between error packets and performance which provides more details about messages vs. 6.1.2

4)      Same comment as previous one for 6.1.3

5)      6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test procedure says about sending single traffic stream between end nodes. But it does not say about traffic stream parameters  (frame size, rate, IPv4 or something else etc.). Would be good to specify it  IMHO (as I mentioned above in 4.2 are not many details in current draft) . Another option could be just comment that these parameters should be defined before testing by customer itself.

6)      6.1.6 and 6.1.7 Reactive and Proactive rates. Same question as in 5. Also here we have to send traffic with some rate and calculate rate at the receiver end.  Sounds that we are looking for NDR here, may be we need to mention it? Because in other tests like 6.4.1 (Procedure Step1) we actually say that traffic should be sent with NDR, so we need to find it during baseline tests.

7)      6.1.8 Network Topology Change Detection Time. Procedure Step 1, should we bring down ANY active node?  Does not matter which one?

8)      6.2 Scalability tests. Shouldn't we put there multidimensional test to find out the maximum numbers of  parameters from 6.2.1-6.2.3 but altogether?

9)      6.4.2  Network Re-Provisioning Time. Again it says about sending bi-directional traffic stream but without any stream details.

Thank you.

SY,
Boris

From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Sarah Banks
Sent: Sunday, August 02, 2015 7:08 AM
To: Marius Georgescu
Cc: bmwg@ietf.org
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Thank you Marius, for your time and inline suggestions. We'll review and touch base soon.

Thanks
Sarah

On Aug 1, 2015, at 1:17 AM, Marius Georgescu <liviumarius-g@is.naist.jp<mailto:liviumarius-g@is.naist.jp>> wrote:

Dear Bhuvan et alia,

Please find attached my inline suggestions for the new draft. They were marked with ###MG .
Thank you for taking care of so many comments.

Best regards,
Marius

From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of Bhuvan (Veryx Technologies)
Sent: Tuesday, July 21, 2015 8:23 PM
To: bmwg@ietf.org<mailto:bmwg@ietf.org>
Cc: 'Anton Basil' <anton.basil@veryxtech.com<mailto:anton.basil@veryxtech.com>>; Tassinari, Mark A <mark.tassinari@hp.com<mailto:mark.tassinari@hp.com>>; vishwas.manral <vishwas.manral@gmail.com<mailto:vishwas.manral@gmail.com>>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Dear BMWG Members,

We have updated the draft (draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00>) about SDN Controller benchmarking addressing comments received in IETF-92 meeting. Thank you very much for providing your valuable comments. The latest draft can be found in draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01>

Summary of Changes:
a.      Updated test setup diagram following the comment from Scott Bradner.
b.      Added recommendations for test topology, test iterations etc., to use for benchmarking.
c.      Provided reference test topologies.
d.      Split Path Provisioning tests into two different tests - Proactive and Reactive Path Provisioning tests.
e.      Provided more clarity on test procedure for some of the tests.
f.       Fixed IETF normative language usage.

We would love to hear any comments and queries on the same.

Thanks,
Authors
<draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 MG.TXT>_______________________________________________
bmwg mailing list
bmwg@ietf.org<mailto:bmwg@ietf.org>
https://www.ietf.org/mailman/listinfo/bmwg