Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

["Bhuvan \(Veryx Technologies\)" <[e-mail]>]

Hi Boris,

 

Thanks for the clarification.

I really like the proposal of having an MD test case for scalability.  But
at the same time, the measurement/result comparison would be difficult when
multiple dimensions vary simultaneously . Also there may be chances that the
result of one dimension could influence the result of other.

 

Best regards,

Bhuvan

 

From: Khasanov Boris [mailto:[e-mail]] 
Sent: 07 August 2015 18:09
To: Bhuvan (Veryx Technologies)
Cc: [e-mail]; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

 

Hi Bhuvan,

 

Thank you.

 

I put my answer below under [BKH].

 

SY,

Boris

 

From: Bhuvan (Veryx Technologies)
[mailto:[e-mail]] 
Sent: Friday, August 07, 2015 2:32 PM
To: Khasanov Boris
Cc: [e-mail]; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

 

Hi Boris,

 

Thank you very much for taking time to review this draft and providing your
valuable comments.  

I've provided my responses in line to your comments.

 

Best regards,

Bhuvan

 

From: bmwg [mailto:[e-mail]] On Behalf Of Khasanov Boris
Sent: 07 August 2015 12:37
To: Sarah Banks
Cc: [e-mail]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

 

Hi Sarah and all,

I am sorry  for jumping late into discussion.

Just few thoughts or my 0,02$:

1)
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01#section-4.2> 4.2 Test Traffic. IMHO, it would be better to say there
explicitly  that in case of performance testing of forwarding plane we are
looking for NDRs for different frame sizes according to RFC2544, if I got it
correct. 

[Bhuvan] I would like to clarify that the draft focusses on defining test
methodology for control plane. So we believe that measuring NDRs for
different frame sizes would not be applicable. But it is important to
perform the defined tests for various frame sizes and traffic type.  So
which we are referring to RFC 2544.


2)
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01#section-6.1.2> 6.1.2 Asynchronous Message Processing Time May be would be
good to add a couple of words for clarification - why it is important to SDN
controller to process such messages as Error or Flow-Removed as quick as
possible. It may sound obvious, but I believe that for many customers such
clarification will be important.  


               [Bhuvan] Sure. We will add some clarification in the
discussion section (2.3.1.2) of
draft-bhuvan-bmwg-sdn-controller-benchmark-term-00 draft.


 


3)
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01#section-6.1.2> 6.1.2 Asynchronous Message Processing Time Would it make
sense to specify explicitly which messages need to be generated? I am not SW
engineer so please correct me if my understanding is wrong, but I think
different async messages may have different processing time on SDN
controller, so results may be different and hard to compare (like comparison
of best results but these results will be collected for different messages
on different controllers).  We also have 6.3.1 test which measures
correlation between error packets and performance which provides more
details about messages vs. 6.1.2

 

4)      Same comment as previous one for 6.1.3

[Bhuvan] I completely agree with you. But the draft defines methodology
generic to all controllers implementation, I'm not sure if we able to
explicitly specify/recommend any particular message. We will provide some
examples for reference.

 

5)      6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test
procedure says about sending single traffic stream between end nodes. But it
does not say about traffic stream parameters  (frame size, rate, IPv4 or
something else etc.). Would be good to specify it  IMHO (as I mentioned
above in 4.2 are not many details in current draft) . Another option could
be just comment that these parameters should be defined before testing by
customer itself.

[Bhuvan] Typically controller provisions the network path for a flow in the
SDN nodes (switches) when it sees the first packet of a flow (reactive).
Once the path is provisioned, all the subsequent packets are forwarded by
the SDN node (e.g., switch) based on the provisioned flow. Here we are
measuring the time that the controller takes to pave the entire path for a
flow in the SDN nodes. So I believe that the packet size or the rate may not
influence the test. 

 

6)      6.1.6 and 6.1.7 Reactive and Proactive rates. Same question as in 5.
Also here we have to send traffic with some rate and calculate rate at the
receiver end.  Sounds that we are looking for NDR here, may be we need to
mention it? Because in other tests like 6.4.1 (Procedure Step1) we actually
say that traffic should be sent with NDR, so we need to find it during
baseline tests.

[Bhuvan] Sure. We will mention the rate.


7)      6.1.8 Network Topology Change Detection Time. Procedure Step 1,
should we bring down ANY active node?  Does not matter which one?


[Bhuvan]  This will not matter for this test.

 

8)      6.2 Scalability tests. Shouldn't we put there multidimensional test
to find out the maximum numbers of  parameters from 6.2.1-6.2.3 but
altogether?

[Bhuvan] Could you please provide some more clarification about this
comment?

[BKH]  I meant the following, in 6.2.1 maximum number of control sessions
will be found, in 6.2.2 network size is measured, in 6.2.3 maximum number of
flow entries. But everything is measured separately (single dimension). In
real life customer may have  big network (many nodes)  so he will have
maximum number of control sessions, maximum number of flows and big network
size simultaneously (multidimensional (MD) case). That is why I personally
would propose to create such MD test case.


9)      6.4.2  Network Re-Provisioning Time. Again it says about sending
bi-directional traffic stream but without any stream details.


[Bhuvan] We will provide the details.

 

Thank you.

 

SY,

Boris

 

From: bmwg [mailto:[e-mail]] On Behalf Of Sarah Banks
Sent: Sunday, August 02, 2015 7:08 AM
To: Marius Georgescu
Cc: [e-mail]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

 

Thank you Marius, for your time and inline suggestions. We'll review and
touch base soon.

 

Thanks

Sarah

 

On Aug 1, 2015, at 1:17 AM, Marius Georgescu <[e-mail]>
wrote:

 

Dear Bhuvan et alia,

 

Please find attached my inline suggestions for the new draft. They were
marked with ###MG .

Thank you for taking care of so many comments.  

 

Best regards,

Marius 

 

 

From: bmwg [ <mailto:[e-mail]> mailto:[e-mail]] On
Behalf Of Bhuvan (Veryx Technologies)
Sent: Tuesday, July 21, 2015 8:23 PM
To:  <mailto:[e-mail]> [e-mail]
Cc: 'Anton Basil' < <mailto:[e-mail]>
[e-mail]>; Tassinari, Mark A <
<mailto:[e-mail]> [e-mail]>; vishwas.manral <
<mailto:[e-mail]> [e-mail]>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance -
Updated Draft Version

 

Dear BMWG Members,

 

We have updated the draft (
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
00> draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00) about SDN Controller
benchmarking addressing comments received in IETF-92 meeting. Thank you very
much for providing your valuable comments. The latest draft can be found in
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01> draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01

 

Summary of Changes:

a.      Updated test setup diagram following the comment from Scott Bradner.

b.      Added recommendations for test topology, test iterations etc., to
use for benchmarking.

c.      Provided reference test topologies.

d.      Split Path Provisioning tests into two different tests - Proactive
and Reactive Path Provisioning tests.

e.      Provided more clarity on test procedure for some of the tests.

f.       Fixed IETF normative language usage.

 

We would love to hear any comments and queries on the same.

 

Thanks,

Authors

<draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01
MG.TXT>_______________________________________________
bmwg mailing list
 <mailto:[e-mail]> [e-mail]
 <https://www.ietf.org/mailman/listinfo/bmwg>
https://www.ietf.org/mailman/listinfo/bmwg