Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

[Khasanov Boris <[e-mail]>]

Hi Bhuvan,

Thank you.

I put my answer below under [BKH].

SY,
Boris

From: Bhuvan (Veryx Technologies) [mailto:[e-mail]]
Sent: Friday, August 07, 2015 2:32 PM
To: Khasanov Boris
Cc: [e-mail]; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Boris,

Thank you very much for taking time to review this draft and providing your valuable comments.
I've provided my responses in line to your comments.

Best regards,
Bhuvan

From: bmwg [mailto:[e-mail]] On Behalf Of Khasanov Boris
Sent: 07 August 2015 12:37
To: Sarah Banks
Cc: [e-mail]<mailto:[e-mail]>
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Sarah and all,
I am sorry  for jumping late into discussion.
Just few thoughts or my 0,02$:

1)      4.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-4.2> Test Traffic. IMHO, it would be better to say there explicitly  that in case of performance testing of forwarding plane we are looking for NDRs for different frame sizes according to RFC2544, if I got it correct.
[Bhuvan] I would like to clarify that the draft focusses on defining test methodology for control plane. So we believe that measuring NDRs for different frame sizes would not be applicable. But it is important to perform the defined tests for various frame sizes and traffic type.  So which we are referring to RFC 2544.
2)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time May be would be good to add a couple of words for clarification - why it is important to SDN controller to process such messages as Error or Flow-Removed as quick as possible. It may sound obvious, but I believe that for many customers such clarification will be important.
               [Bhuvan] Sure. We will add some clarification in the discussion section (2.3.1.2) of draft-bhuvan-bmwg-sdn-controller-benchmark-term-00 draft.


3)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time Would it make sense to specify explicitly which messages need to be generated? I am not SW engineer so please correct me if my understanding is wrong, but I think different async messages may have different processing time on SDN controller, so results may be different and hard to compare (like comparison of best results but these results will be collected for different messages on different controllers).  We also have 6.3.1 test which measures correlation between error packets and performance which provides more details about messages vs. 6.1.2



4)      Same comment as previous one for 6.1.3
[Bhuvan] I completely agree with you. But the draft defines methodology generic to all controllers implementation, I'm not sure if we able to explicitly specify/recommend any particular message. We will provide some examples for reference.



5)      6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test procedure says about sending single traffic stream between end nodes. But it does not say about traffic stream parameters  (frame size, rate, IPv4 or something else etc.). Would be good to specify it  IMHO (as I mentioned above in 4.2 are not many details in current draft) . Another option could be just comment that these parameters should be defined before testing by customer itself.
[Bhuvan] Typically controller provisions the network path for a flow in the SDN nodes (switches) when it sees the first packet of a flow (reactive). Once the path is provisioned, all the subsequent packets are forwarded by the SDN node (e.g., switch) based on the provisioned flow. Here we are measuring the time that the controller takes to pave the entire path for a flow in the SDN nodes. So I believe that the packet size or the rate may not influence the test.



6)      6.1.6 and 6.1.7 Reactive and Proactive rates. Same question as in 5. Also here we have to send traffic with some rate and calculate rate at the receiver end.  Sounds that we are looking for NDR here, may be we need to mention it? Because in other tests like 6.4.1 (Procedure Step1) we actually say that traffic should be sent with NDR, so we need to find it during baseline tests.
[Bhuvan] Sure. We will mention the rate.
7)      6.1.8 Network Topology Change Detection Time. Procedure Step 1, should we bring down ANY active node?  Does not matter which one?
[Bhuvan]  This will not matter for this test.



8)      6.2 Scalability tests. Shouldn't we put there multidimensional test to find out the maximum numbers of  parameters from 6.2.1-6.2.3 but altogether?

[Bhuvan] Could you please provide some more clarification about this comment?

[BKH]  I meant the following, in 6.2.1 maximum number of control sessions will be found, in 6.2.2 network size is measured, in 6.2.3 maximum number of flow entries. But everything is measured separately (single dimension). In real life customer may have  big network (many nodes)  so he will have maximum number of control sessions, maximum number of flows and big network size simultaneously (multidimensional (MD) case). That is why I personally would propose to create such MD test case.

9)      6.4.2  Network Re-Provisioning Time. Again it says about sending bi-directional traffic stream but without any stream details.
[Bhuvan] We will provide the details.


Thank you.

SY,
Boris

From: bmwg [mailto:[e-mail]]<mailto:[mailto:[e-mail]]> On Behalf Of Sarah Banks
Sent: Sunday, August 02, 2015 7:08 AM
To: Marius Georgescu
Cc: [e-mail]<mailto:[e-mail]>
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Thank you Marius, for your time and inline suggestions. We'll review and touch base soon.

Thanks
Sarah

On Aug 1, 2015, at 1:17 AM, Marius Georgescu <[e-mail]<mailto:[e-mail]>> wrote:

Dear Bhuvan et alia,

Please find attached my inline suggestions for the new draft. They were marked with ###MG .
Thank you for taking care of so many comments.

Best regards,
Marius


From: bmwg [mailto:[e-mail]] On Behalf Of Bhuvan (Veryx Technologies)
Sent: Tuesday, July 21, 2015 8:23 PM
To: [e-mail]<mailto:[e-mail]>
Cc: 'Anton Basil' <[e-mail]<mailto:[e-mail]>>; Tassinari, Mark A <[e-mail]<mailto:[e-mail]>>; vishwas.manral <[e-mail]<mailto:[e-mail]>>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Dear BMWG Members,

We have updated the draft (draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00>) about SDN Controller benchmarking addressing comments received in IETF-92 meeting. Thank you very much for providing your valuable comments. The latest draft can be found in draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01>

Summary of Changes:
a.      Updated test setup diagram following the comment from Scott Bradner.
b.      Added recommendations for test topology, test iterations etc., to use for benchmarking.
c.      Provided reference test topologies.
d.      Split Path Provisioning tests into two different tests - Proactive and Reactive Path Provisioning tests.
e.      Provided more clarity on test procedure for some of the tests.
f.       Fixed IETF normative language usage.

We would love to hear any comments and queries on the same.

Thanks,
Authors
<draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 MG.TXT>_______________________________________________
bmwg mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/bmwg