Re: [bmwg] comments on draft-bhuvan-bmwg-of-controller-benchmarking-01

["Bhuvaneswaran Vengainathan" <[e-mail]>]

Hi Al,

Thank you very much for the detailed review and providing your valuable
comments.
Your comments are very helpful. I might need some clarifications on some of
your comments which I will discuss internally with others authors and get
back to you.

Best regards,
Bhuvan

> -----Original Message-----
> From: MORTON, ALFRED C (AL) [mailto:[e-mail]]
> Sent: Thursday, October 23, 2014 2:00 AM
> To: [e-mail]; draft-bhuvan-bmwg-of-controller-
> [e-mail]
> Subject: comments on draft-bhuvan-bmwg-of-controller-benchmarking-01
> 
> Hi Bhuvan, Anton, Vishwas, and Mark,
> 
> Thanks for preparing a very complete and interesting draft!
> 
> My comments on the draft are dispersed throughout the text below, all
> prefaced by "ACM:"
> 
> regards,
> Al
> (as participant)
> 
> 
>      Benchmarking Methodology for SDN Controller Performance
>         draft-bhuvan-bmwg-of-controller-benchmarking-01
> 
> ...
> 1. Introduction
> 
>    This document provides generic metrics and methodologies for
>    benchmarking SDN controller performance. An SDN controller may
>    support many northbound and southbound protocols, implement wide
>    range of applications and work as standalone or as a group to
>    achieve the desired functionality. This document considers an SDN
>    controller as a black box, regardless of design and implementation.
>    The tests defined in the document can be used to benchmark various
>    controller designs for performance, scalability, reliability and
>    security independent of northbound and southbound protocols. These
>    tests can be performed on an SDN controller running as a virtual
>    machine (VM) instance or on a bare metal server. This document is
>    intended for those who want to measure the SDN controller
>    performance as well as compare various SDN controllers performance.
> 
>    Conventions used in this document
> 
>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
> this
>    document are to be interpreted as described in RFC 2119.
> 
> 2. Terminology
> 
> ACM:
> Let's try to be consistent with other efforts that have also defined
terms.  For
> example, the IRTF SDNRG has an approved draft now, with many related
> terms:
> http://tools.ietf.org/html/draft-irtf-sdnrg-layer-terminology-04
> They define terms like SDN and Interface in the SDN context.
> The draft also provides an SDN Architecture in Figure 1, showing 2
different
> types of Southbound Interfaces (Control and Management).
> 
>    SDN Node:
>       An SDN node is a physical or virtual entity that forwards
>       data in a software defined environment.
> 
>    Flow:
>       A flow is a traffic stream having same source and destination
>       address. The address could be MAC or IP or combination of both.
> ACM:
> This could be closer to the definition of a microflow, see
> http://tools.ietf.org/html/rfc4689#section-3.1.5
> 
> 
>    Learning Rate:
>       The rate at which the controller learns the new source addresses
>       from the received traffic without dropping.
> ACM:
> I suggest to leave out "without dropping", to give a more general metric.
> Using this definition, we could define the "lossless" or "reliable"
Learning rate
> where the additional condition of no messages dropped applies.
> 
> 
>    Controller Forwarding Table:
>       A controller forwarding table contains flow records for the flows
>       configured in the data path.
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 3]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
>    Northbound Interface:
>       Northbound interface is the application programming interface
>       provided by the SDN controller for communication with SDN
>       services and applications.
> ACM:
> http://tools.ietf.org/html/draft-irtf-sdnrg-layer-terminology-04#page-7
> Figure 1 doesn't show the Northbound interface (but it doesn't
specifically
> show the boundaries of the controller, either...
> 
>    Southbound Interface:
>       Southbound interface is the application programming interface
>       provided by the SDN controller for communication with the SDN
>       nodes.
> 
>    Proactive Flow Provisioning:
>       Proactive flow provisioning is the pre-provisioning of flow
>       entries into the controller's forwarding table through
>       controller's northbound interface or management interface.
> 
>    Reactive Flow Provisioning:
>       Reactive flow provisioning is the dynamic provisioning of flow
>       entries into the controller's forwarding table based on traffic
>       forwarded by the SDN nodes through controller's southbound
>       interface.
> 
>    Path:
>       A path is the route taken by a flow while traversing from a source
>       node to destination node.
> ACM:
> "route" seems unclear, we want to say something about the nodes
> traversed.
> There are lots of definitions of path, we could adapt the one from from
> RFC2330 (below), or other source if you want:
>    path A sequence of the form < h0, l1, h1, ..., ln, hn >, where n >=
>         0, each hi is a host, each li is a link between hi-1 and hi,
>         each h1...hn-1 is a router.  A pair <li, hi> is termed a 'hop'.
>         In an appropriate operational configuration, the links and
>         routers in the path facilitate network-layer communication of
>         packets from h0 to hn.  Note that path is a unidirectional
>         concept.
> 
> 
>    Standalone Mode:
>       Single controller handling all control plane functionalities.
> 
>    Cluster/Redundancy Mode:
>       Group of controllers handling all control plane functionalities .
> 
> ACM: for the Mode definitions above:
> The Group case should indicate possibilities for how the group shares the
> control responsibilities: shared load, separate loads, active/standby,
etc.  The
> name Cluster/Redundancy could be any of these types - maybe define each
> separately. For the group case, How are the Management Plane functions
> divided? should add this aspect.
> 
>    Synchronous Message:
>       Any message from the SDN node that triggers a response message
>       from the controller e.g., Keepalive request and response message,
>       flow setup request and response message etc.,
> 
> ACM:
> "synchronous" seems like the wrong adjective here.  Did this term come
> from one of the implementations?  Seems like "request-response" message
> or "response required" message is more exact, but that's just the idealist
> commenting...
> 
> 3. Scope
> 
>    This document defines a number of tests to measure the networking
>    aspects of SDN controllers. These tests are recommended for
>    execution in lab environments rather than in real time deployments.
> 
> ACM:
> s/measure the/measure the performance of/ suggest s/networking/control
> and management/ (or just control)
> 
> 
> 
> 
> 
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 4]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
> 4. Test Setup
> 
>    The tests defined in this document enable measurement of SDN
>    controller's performance in Standalone mode and Cluster mode. This
>    section defines common reference topologies that are later referred
>    to in individual tests.
> 
> ACM:
> In the network cases below (4.1-4.4), we should probably show the network
> path more explicitly (since we went to the trouble to define it above)
> between the nodes.  So here the path would be Node1, link, Node2, link, .
. .
> Noden
> 
>                   ----------      ----------        ----------
>                  |   SDN    |____|   SDN    |__..__|   SDN    |
>                  |  Node 1  |    |  Node 2  |      |  Node n  |
>                   ----------      ----------        ----------
> 
> 
> 4.1 SDN Network - Controller working in Standalone Mode
> 
>                           --------------------
>                          |  SDN Applications  |
>                           --------------------
>                                    |
>                                    | (Northbound interface)
>                          -----------------------
>                         |     SDN Controller    |
>                         |          (DUT)        |
>                          -----------------------
>                                    | (Southbound interface)
>                                    |
>                        ---------------------------
>                       |            |              |
>                   ----------    ----------    ----------
>                  |   SDN    |  |   SDN    |..|   SDN    |
>                  |  Node 1  |  |  Node 2  |  |  Node n  |
>                   ----------    ----------    ----------
> 
>                                   Figure 1
> 
> 4.2 SDN Network - Controller working in Cluster Mode
> 
>                           --------------------
>                          |  SDN Applications  |
>                           --------------------
>                                    |
>                                    | (Northbound interface)
>         ---------------------------------------------------------
>        |  ------------------             ------------------      |
>        | | SDN Controller 1 | <--E/W--> | SDN Controller n |     |
>        |  ------------------             ------------------      |
>         ---------------------------------------------------------
>                                    | (Southbound interface)
>                                    |
>                        ---------------------------
>                       |            |              |
>                   ----------    ----------    ----------
>                  |   SDN    |  |   SDN    |..|   SDN    |
>                  |  Node 1  |  |  Node 2  |  |  Node n  |
>                   ----------    ----------    ----------
> 
>                                   Figure 2
> ACM:
> Does this apply to shared control and active/standby ?
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 5]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
> 
> 4.3 SDN Network with Traffic Endpoints (TE) - Controller working in
>     Standalone Mode
> 
>                           --------------------
>                          |  SDN Applications  |
>                           --------------------
>                                    |
>                                    | (Northbound interface)
>                          -----------------------
>                         |  SDN Controller (DUT) |
>                          -----------------------
>                                    | (Southbound interface)
>                                    |
>                        ---------------------------
>                       |            |              |
>                   ----------    ----------    ----------
>                  |   SDN    |  |   SDN    |..|   SDN    |
>                  |  Node 1  |  |  Node 2  |  |  Node n  |
>                   ----------    ----------    ----------
>                       |                           |
>                 --------------             --------------
>                |   Traffic    |           |   Traffic    |
>                | Endpoint TP1 |           | Endpoint TP2 |
>                 --------------             --------------
> 
>                                   Figure 3
> 
> 4.4 SDN Network with Traffic Endpoints (TE) - Controller working in
>     Cluster Mode
> 
>                           --------------------
>                          |  SDN Applications  |
>                           --------------------
>                                    |
>                                    | (Northbound interface)
>         ---------------------------------------------------------
>        |  ------------------             ------------------      |
>        | | SDN Controller 1 | <--E/W--> | SDN Controller n |     |
>        |  ------------------             ------------------      |
>         ---------------------------------------------------------
>                                    |
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 6]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
>                                    | (Southbound interface)
>                                    |
>                        ---------------------------
>                       |            |              |
>                   ----------    ----------    ----------
>                  |   SDN    |  |   SDN    |..|   SDN    |
>                  |  Node 1  |  |  Node 2  |  |  Node n  |
>                   ----------    ----------    ----------
>                       |                           |
>                 --------------             --------------
>                |   Traffic    |           |   Traffic    |
>                | Endpoint TP1 |           | Endpoint TP2 |
>                 --------------             --------------
> 
>                                   Figure 4
> 
> 
> 4.5 SDN Node with Traffic Endpoints (TE) - Controller working in
>     Standalone Mode
>                           --------------------
>                          |  SDN Applications  |
>                           --------------------
>                                    |
>                                    | (Northbound interface)
>                          -----------------------
>                         |     SDN Controller    |
>                         |          (DUT)        |
>                          -----------------------
>                                    | (Southbound interface)
>                                    |
>                                ----------
>                        -------|   SDN    |---------
>                       |       |  Node 1  |         |
>                       |        ----------          |
>                   ----------                  ----------
>                  | Traffic  |                | Traffic  |
>                  | Endpoint |                | Endpoint |
>                  |   TP1    |                |   TP2    |
>                   ----------                  ----------
> 
>                                   Figure 5
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 7]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
> 4.6 SDN Node with Traffic Endpoints (TE) - Controller working in Cluster
>     Mode
> 
>                           --------------------
>                          |  SDN Applications  |
>                           --------------------
>                                    |
>                                    | (Northbound interface)
>         ---------------------------------------------------------
>        |  ------------------             ------------------      |
>        | | SDN Controller 1 | <--E/W--> | SDN Controller n |     |
>        |  ------------------             ------------------      |
>         ---------------------------------------------------------
>                                    | (Southbound interface)
>                                    |
>                                ----------
>                        -------|   SDN    |---------
>                       |       |  Node 1  |         |
>                       |        ----------          |
>                   ----------                  ----------
>                  | Traffic  |                | Traffic  |
>                  | Endpoint |                | Endpoint |
>                  |   TP1    |                |   TP2    |
>                   ----------                  ----------
> 
>                                   Figure 6
> 
> 5. Test Considerations
> 
> 5.1 Network Topology
> 
>    The network SHOULD be deployed with SDN nodes interconnected in
>    either fully meshed, tree or linear topology. Care should be taken
>    to make sure that the loop prevention mechanism is enabled either in
>    the SDN controller or in the network. To get complete performance
>    characterization of SDN controller, it is recommended that the
>    controller be benchmarked for many network topologies. These network
>    topologies can be deployed using real hardware or emulated in
>    hardware platforms.
> 
> 5.2 Test Traffic
> 
>    Test traffic can be used to notify the controller about the arrival
>    of new flows or generate notifications/events towards controller.
>    In either case, it is recommended that at least five different frame
>    sizes and traffic types be used, depending on the intended network
>    deployment.
> 
> ACM:
> Single size tests?  (should be "yes")
> We should recommend the default sizes here or reference another set.
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 8]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
> 5.3 Connection Setup
> 
>    There may be controller implementations that support
>    unencrypted and encrypted network connections with SDN nodes.
>    Further, the controller may have backward compatibility with SDN
>    nodes running older versions of southbound protocols. It is
>    recommended that the controller performance be measured with the
>    applicable connection setup methods.
> 
>    1. Unencrypted connection with SDN nodes, running same protocol
>       version.
>    2. Unencrypted connection with SDN nodes, running
>       different (previous) protocol versions.
>    3. Encrypted connection with SDN nodes,running same protocol version
>    4. Encrypted connection with SDN nodes, running
>       different (previous)protocol versions.
> 
> ACM:
> suggest
> s/previous/current and older/
> 
> 5.4 Measurement Accuracy
> 
>    The measurement accuracy depends on the
>    point of observation where the indications are captured. For example,
>    the notification can be observed at the ingress or egress point of
>    the SDN node. If it is observed at the egress point of the SDN node,
>    the measurement includes the latency within the SDN node also. It is
>    recommended to make observation at the ingress point of the SDN node
>    unless it is explicitly mentioned otherwise in the individual test.
> 
> ACM:
> This is really about specificity of measurement points.
> The accuracy of results-reporting depends on the measurement point
> specifications, but there are lots of other factors affecting accuracy.
> I suggest calling this section
> "Measurement Point Specification and Recommendation"
> 
> 
> 5.5 Real World Scenario
> 
>    Benchmarking tests discussed in the document are
>    to be performed on a "black-box" basis, relying solely on
>    measurements observable external to the controller. The network
>    deployed and the test parameters should be identical to the
>    deployment scenario to obtain value added measures.
> 
> ACM:
> suggest:
> ... to obtain measurements with the greatest value.
> 
> 6. Test Reporting
> 
>    Each test has a reporting format which is specific to individual
>    test. In addition, the following configuration parameters SHOULD be
>    reflected in the test report.
>    1. Controller name and version
>    2. Northbound protocols and version
>    3. Southbound protocols and version
>    4. Controller redundancy mode (Standalone or Cluster Mode)
>    5. Connection setup (Unencrypted or Encrypted)
>    6. Network Topology (Mesh or Tree or Linear)
>    7. SDN Node Type (Physical or Virtual or Emulated)
>    8. Number of Nodes
>    9. Number of Links
>    10. Test Traffic Type
> 
> ACM:
> I think we may need some more HW specifications here.
> check-out:
> https://tools.ietf.org/html/draft-morton-bmwg-virtual-net-01#section-3
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015               [Page 9]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
> 7. Benchmarking Tests
> 
> 7.1 Performance
> 
> 7.1.1 Network Topology Discovery Time
> 
> ACM:
> This is a good Benchmark.  One small detail is that we usually present the
> Benchmark Definitions separately from the test procedures - it makes it
> easier to understand what will be quantified in a section with all the
> Benchmark definitions side by side.  More comments below.
> 
>    Objective:
>       To measure the time taken to discover the network topology- nodes
>       and its connectivity by a controller, expressed in milliseconds.
> 
>    Setup Parameters:
>       The following parameters MUST be defined:
> 
>       Network setup parameters:
>       Number of nodes (N) - Defines the number of nodes present in the
>       defined network topology
> ACM:
> suggest:
>       Topology: clear specification (e.g., full mesh) or diagram.
> ------
> 
> ACM:
> Latency on the links between nodes will affect the result, right?
> Perhaps this should be measured and reported, too.
> ------
> 
>       Test setup parameters:
>       Test Iterations (Tr) - Defines the number of times the test needs
>       to be repeated. The recommended value is 3.
>       Test Interval (To)- Defines the maximum time for the test to
>       complete, expressed in milliseconds.
> ACM:
> For un-successful discovery iterations, how are the results reported?
> 
>       Test Setup:
>       The test can use one of the test setup described in section 4.1
>       and 4.2 of this document.
> 
>    Prerequisite:
>       1.  The controller should support network discovery.
> ACM:
> . . . MUST support network discovery???
> 
>       2.  Tester should be able to retrieve the discovered topology
> ACM:
> s/should/SHOULD/
>           information either through controller's management interface
>           or northbound interface.
> ACM: add
> . . . to determine if the discovery was successful and complete.
> 
>    Procedure:
>       1.  Initialize the controller - network applications, northbound
>           and southbound interfaces.
>       2.  Deploy the network with the given number of nodes using mesh
>           or linear topology.
>       3.  Initialize the network connections between controller and
>           network nodes.
> ACM:
> So, the controller starts out knowing all the nodes it controls, that
makes
> sense with Topology discovery.
> 
>       4.  Record the time for the first discovery message exchange
>           between the controller and the network node (Tm1).
>       5.  Query the controller continuously for the discovered network
>           topology information and compare it with the deployed network
>           topology information.
>       6.  Stop the test when the discovered topology information is
>           matching with the deployed network topology or the expiry of
>           test interval (To).
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015              [Page 10]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
>       7.  Record the time last discovery message exchange between the
>           controller and the network node (Tmn) when the test completed
>           successfully.
> ACM:
> . . . successfully (e.g., the topology matches).
> 
>    Note: While recording the Tmn value, it is recommended that the
>          messages that are used for aliveness check or session
>          management be ignored.
> 
>    Measurement:
>       Topology Discovery Time Tr1 = Tmn-Tm1.
> 
>                                         Tr1 + Tr2 + Tr3 .. Trn
>       Average Topology Discovery Time = -----------------------
>                                         Total Test Iterations
> 
>    Note:
>       1. To increase the certainty of measured result, it is
> ACM:
> s/certainty of/confidence in/
> 
>          recommended that this test be performed several times with
>          same number of nodes using same topology.
>       2. To get the full characterization of a controller's topology
>          discovery functionality
>          a. Perform the test with varying number of nodes using same
>             topology
>          b. Perform the test with same number of nodes using different
>             topologies.
> 
>    Reporting Format:
>       The Topology Discovery Time results SHOULD be reported in the
>       format of a table, with a row for each iteration. The last row of
>       the table indicates the average Topology Discovery Time.
> 
>       If this test is repeated with varying number of nodes over the
>       same topology, the results SHOULD be reported in the form of a
>       graph. The X coordinate SHOULD be the Number of nodes (N), the
>       Y coordinate SHOULD be the average Topology Discovery Time.
> ACM:
> nicely done, and very traditional.
> 
>       If this test is repeated with same number of nodes over different
>       topologies,the results SHOULD be reported in the form of a graph.
>       The X coordinate SHOULD be the Topology Type, the Y coordinate
>       SHOULD be the average Topology Discovery Time.
> 
> ACM:
> many of the comments above apply to the sections below, I won't repeat
> them.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015              [Page 11]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
> 7.1.2 Synchronous Message Processing Time
> 
>    Objective:
>       To measure the time taken by the controller to process a
>       synchronous message, expressed in milliseconds.
> 
>    Setup Parameters:
>       The following parameters MUST be defined:
> 
>       Network setup parameters:
>       Number of nodes (N) - Defines the number of nodes present in the
>       defined network topology
> 
>       Test setup parameters:
>       Test Iterations (Tr) - Defines the number of times the test needs
>       to be repeated. The recommended value is 3.
>       Test Duration (Td) - Defines the duration of test iteration,
>       expressed in seconds. The recommended value is 5 seconds.
> 
>       Test Setup:
>       The test can use one of the test setup described in section 4.1
>       and 4.2 of this document.
> 
>    Prerequisite:
>       1. The controller should have completed the network topology
>          discovery for the connected nodes.
> 
>    Procedure:
>       1. Generate a synchronous message from every connected nodes one
>          at a time and wait for the response before generating the
>          next message.
> ACM:
> So this is serial message processing time.
> We may want to distinguish this in the name of the Benchmark.
> -------
> 
> ACM:
> I don't see how the loss of a request message would be handled.
> This needs to be mentioned somewhere.  For example I suppose a request
> sender will time-out and re-send the request, but then we need to know
> that time-out.  Time-outs have a large impact on the average for that
> iteration - perhaps there should be a way to count the re-transmitted
> requests. There's definitely an issue here.
> -------
> 
> 
>       2. Record total number of messages sent to the controller by all
>          nodes (Ntx) and the responses received from the
>          controller (Nrx) within the test duration (Td).
> 
> ACM:
> So this is a fixed duration test, and the number of successful responses
> completed determines the average request response time.
> 
>    Measurement:
>                                                   Td
>       Synchronous Message Processing Time Tr1 = ------
>                                                   Nrx
> 
>                                                    Tr1 + Tr2 + Tr3..Trn
>       Average Synchronous Message Processing Time= --------------------
>                                                   Total Test Iterations
> 
> 
> 
> 
> 
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015              [Page 12]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
>    Note:
>       1. The above test measures the controller's message processing
>          time at lower traffic rate. To measure the controller's
>          message processing time at full connection rate, apply the
>          same measurement equation with the Td and Nrx values obtained
>          from Synchronous Message Processing Rate test
>          (defined in Section 7.1.3).
>       2. To increase the certainty of measured result, it is
>          recommended that this test be performed several times with
>          same number of nodes using same topology.
>       3. To get the full characterization of a controller's synchronous
>          message processing time
>          a. Perform the test with varying number of nodes using same
>             topology
>          b. Perform the test with same number of nodes using different
>             topologies.
> 
>    Reporting Format:
>       The Synchronous Message Processing Time results SHOULD be
>       reported in the format of a table with a row for each iteration.
>       The last row of the table indicates the average Synchronous
>       Message Processing Time.
> 
>       The report should capture the following information in addition
>       to the configuration parameters captured in section 6.
>       - Offered rate (Ntx)
> 
>       If this test is repeated with varying number of nodes with same
>       topology, the results SHOULD be reported in the form of a graph.
>       The X coordinate SHOULD be the Number of nodes (N), the
>       Y coordinate SHOULD be the average Synchronous Message Processing
>       Time.
> 
>       If this test is repeated with same number of nodes using
>       different topologies, the results SHOULD be reported in the form
>       of a graph. The X coordinate SHOULD be the Topology Type, the
>       Y coordinate SHOULD be the average Synchronous Message Processing
>       Time.
> 
> 7.1.3 Synchronous Message Processing Rate
> 
>    Objective:
>       To measure the maximum number of synchronous messages (session
>       aliveness check message, new flow arrival notification
>       message etc.) a controller can process within the test duration,
>       expressed in messages processed per second.
> 
> ACM:
> even when the controller is dropping messages?
> (this is kind of the Benchmark definition above, so I'm asking for
clarification)
> 
> 
> 
> 
> 
> Bhuvan, et al.            Expires March 26, 2015              [Page 13]
> 
> Internet Draft    SDN Controller Benchmarking Methodology    March 2015
> 
> 
>    Setup Parameters:
>       The following parameters MUST be defined:
> 
>       Network setup parameters:
>       Number of nodes (N) - Defines the number of nodes present in the
>       defined network topology.
> 
>       Test setup parameters:
>       Test Iterations (Tr) - Defines the number of times the test needs
>       to be repeated. The recommended value is 3.
>       Test Duration (Td) - Defines the duration of test iteration,
>       expressed in seconds. The recommended value is 5 seconds.
> 
>       Test Setup:
>       The test can use one of the test setup described in section 4.1
>       and 4.2 of this document.
> 
>    Prerequisite:
>       1. The controller should have completed the network topology
>          discovery for the connected nodes.
> 
>    Procedure:
>       1. Generate synchronous messages from all the connected nodes
>          at the full connection capacity for the Test Duration (Td).
> ACM:
> I think we need to add detail on the connection capacity from each node to
> the controller.  Is it a shared link with an aggregation point?  Or, do
these
> control connections use traffic management, and we are talking about the
> capacity of a virtual pipe, not PHY.?
> 
>       2. Record total number of messages sent to the controller by all
>          nodes (Ntx) and the responses received from the
>          controller (Nrx) within the test duration (Td).
> 
> ACM:
> We have to distinguish the lossy case, where Ntx not.equal Nrx.
> ----------
> 
>    Measurement:
>                                                  Nrx
>       Synchronous Message Processing Rate Tr1 = -----
>                                                  Td
>                                                    Tr1 + Tr2 + Tr3..Trn
>       Average Synchronous Message Processing Rate= --------------------
>                                                   Total Test Iterations
> 
> ACM:
> and I think we want a version of this for lossless operation.
> perhaps another case with loss ratio measured would also be useful.
> ALSO, I think this comment to recognize Loss conditions may apply to the
> procedures that follow.
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 
> ACM:
> A lot of good Benchmark tests follow, but I will stop here since i've
already
> made quite a few comments.
> 
> I think the 3x3 matrix is helpful in the draft, because there are so many
> benchmarks described.
> 
> From the user perspective, the Scalability metrics affect the reliability
of the
> system as they would perceive it. For example a system operating at max
> capacity would block further requests until space is available, so I can
make a
> case that scale influences reliability (but so does capacity engineering)
> 
> EOT.