Re: [bmwg] Comments on sec 2.3.1 of draft-ietf-bmwg-sdn-controller-benchmark-term-01

["Bhuvan \(Veryx\)" <[e-mail]>]

Hi Al,

Thank you for taking time to review the draft and providing your first set
of valuable comments.
Please find below my response with tag [Bhuvan].

I look forward to your further comments.

Best regards,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[e-mail]] On Behalf Of MORTON, ALFRED C (AL)
Sent: 25 June 2016 04:37
To: [e-mail]
Subject: [bmwg] Comments on sec 2.3.1 of
draft-ietf-bmwg-sdn-controller-benchmark-term-01

Authors,

I have several comments and suggestions on 2.3.1, so I send them to you
without waiting to go further into the Scalability section. See [ACM] below.

Al
(as a participant)

2.3. Benchmarking Terms

   This section defines metrics for benchmarking the SDN controller.
   The procedure to perform the defined metrics is defined in the
   accompanying methodology document.

2.3.1. Performance

2.3.1.1. Network Topology Discovery Time

Definition:
   To measure the time taken to discover the network topology - nodes
   and links by a controller.
[ACM]
The time taken by a controller to determine the complete network topology,
defined as the interval starting with the first discovery message from the
controller(s) at its Southbound interface, ending with all features of the
static topology determined (and expressed in some form in the
controller(s)).
[Bhuvan] I agree. We will reword the definition.

Discussion:
   Network topology discovery is key for the SDN controller to
   provision and manage the network. So it is important to measure how
   quickly the controller discovers the topology to learn the current
   network state. This benchmark is obtained by presenting a network
   topology (Tree, Mesh or Linear) with the given number of nodes to
   the controller and wait for the discovery process to complete .It is
   expected that the controller supports network discovery mechanism
   and uses protocol messages for its discovery process.
[ACM] for metrics like this, we need to make the input parameters clear:
Topology: Leaf-Spine, or linear,
Topology Size: (e.g., number of leaf switches, number of spine switches)
[Bhuvan]I agree. Can we capture this detail as part of reporting?




Bhuvan, et al         Expires September 19, 2016               [Page 9]


Internet-Draft SDN Controller Benchmarking Terminology       March 2016


Measurement Units:
   milliseconds

See Also:
   None


2.3.1.2. Asynchronous Message Processing Time

Definition:
   To measure the time taken by the controller to process an
   asynchronous message.
[ACM]
need to anchor the beginning and end of the interval (as above).
suggestions?
[Bhuvan] Will it be fine if we reword the definition as - "To measure the
time taken by the controller to process an asynchronous message, defined as
the interval starting after the discovery of all the devices by the
controller at its Southbound interface, ending with the given test duration"

Discussion:
   For SDN to support dynamic network provisioning, it is important to
   measure how quickly the controller responds to an event triggered
   from the network. The event could be any notification messages
   generated by an Network Device upon arrival of a new flow, link down
   etc. This benchmark is obtained by sending asynchronous messages
   from every connected Network Devices one at a time for the defined
   test duration. This test assumes that the controller will respond to
   the received asynchronous message.
[ACM] regarding the assumption above:
When operating at the limits of controller capacity, response messages may
be extremely delayed or lost.  (IOW, the assumption doesn't hold) 
[Bhuvan] I agree. But this test ignores the transaction if the response for
the corresponding request is lost or delayed beyond the test duration. 

Measurement Units:
   milliseconds

See Also:
   None


2.3.1.3. Asynchronous Message Processing Rate

Definition:
   To measure the maximum number of asynchronous messages that a
   controller can process within the test duration.

Discussion:
   As SDN assures flexible network and agile provisioning, it is
   important to measure how many network events that the controller can
   handle at a time. This benchmark is obtained by sending asynchronous
   messages from every connected Network Devices at full connection
   capacity for the given test duration. This test assumes that the
   controller will respond to all the received asynchronous messages.
[ACM] regarding the assumption above:
When operating at the limits of controller capacity, response messages may
be extremely delayed or lost.  (IOW, the assumption doesn't hold)
[Bhuvan] I agree. But the test ignores the responses received beyond the
test duration.
 

Measurement Units:
   Messages processed per second.



Bhuvan, et al         Expires September 19, 2016              [Page 10]


Internet-Draft SDN Controller Benchmarking Terminology       March 2016


See Also:
   None

2.3.1.4. Reactive Path Provisioning Time

Definition:
   The time taken by the controller to setup a path reactively between
   source and destination node, expressed in milliseconds.

Discussion:
   As SDN supports agile provisioning, it is important to measure how
   fast that the controller provisions an end-to-end flow in the
   dataplane. The benchmark is obtained by sending traffic from a
   source endpoint to the destination endpoint, finding the time
   difference between the first and the last flow provisioning message
   exchanged between the controller and the Network Devices for the
   traffic path.
[ACM]
Here we need input parameters, such as number of nodes in the path,
background load on the controller and nodes/switches.

THere's also a comparable dataplane measurement, too (First Packet Latency)
[Bhuvan] I agree. We will revisit this section.

Measurement Units:
   milliseconds.

See Also:
   None

2.3.1.5. Proactive Path Provisioning Time

Definition:
   The time taken by the controller to setup a path proactively between
   source and destination node, expressed in milliseconds.

Discussion:
   For SDN to support pre-provisioning of traffic path from
   application, it is important to measure how fast that the controller
   provisions an end-to-end flow in the dataplane. The benchmark is
   obtained by provisioning a flow on controller's northbound interface
   for the traffic to reach from a source to a destination endpoint,
   finding the time difference between the first and the last flow
   provisioning message exchanged between the controller and the
   Network Devices for the traffic path.

Measurement Units:
   milliseconds.

See Also:
   None




Bhuvan, et al         Expires September 19, 2016              [Page 11]


Internet-Draft SDN Controller Benchmarking Terminology       March 2016


2.3.1.6. Reactive Path Provisioning Rate

Definition:
   Measure the maximum number of independent paths a controller can
   concurrently establish between source and destination nodes
   reactively within the test duration, expressed in paths per second.

Discussion:
   For SDN to support agile traffic forwarding, it is important to
   measure how many end-to-end flows that the controller could setup in
   the dataplane. This benchmark is obtained by sending traffic each
   with unique source and destination pairs from the source Network
   Device and determine the number of frames received at the
   destination Network Device.
[ACM]
Here we need input parameters, such as number of nodes in the path,
background load on the controller and nodes/switches.
[Bhuvan] Would this be fine if we record the details as part of reporting?

Also, this measurement seems to make the assumption that the controller does
not loose or discard requests, which may not hold when approaching the Max
Rate.
[Bhuvan] This test does not make such assumption. Could you please explain
this further?

Measurement Units:
   Paths provisioned per second.

See Also:
   None

2.3.1.7. Proactive Path Provisioning Rate

Definition:
   Measure the maximum number of independent paths a controller can
   concurrently establish between source and destination nodes
   proactively within the test duration, expressed in paths per second.

Discussion:
   For SDN to support pre-provisioning of traffic path for a larger
   network from the application, it is important to measure how many
   end-to-end flows that the controller could setup in the dataplane.
   This benchmark is obtained by sending traffic each with unique
   source and destination pairs from the source Network Device. Program
   the flows on controller's northbound interface for traffic to reach
   from each of the unique source and destination pairs and determine
   the number of frames received at the destination Network Device.
[ACM]
Here we need input parameters, such as number of nodes in the path,
background load on the controller and nodes/switches.
Also, this measurement seems to make the assumption that the controller does
not loose or discard requests, which may not hold when approaching the Max
Rate.

Measurement Units:
   Paths provisioned per second.

See Also:
   None

2.3.1.8. Network Topology Change Detection Time

Definition:

   The amount of time required for the controller to detect any changes
   in the network topology.
[ACM]
The "any" in this definition is cause for concern, Need some Input
parameters, such as:
Number or changes, categorized by number of links removed, number os
switches/nodes remove.
[Bhuvan] The term "any" here refers to any network events e.g., link down or
switch down. 

Discussion:
   In order to for the controller to support fast network failure
   recovery, it is critical to measure how fast the controller is able
   to detect any network-state change events. This benchmark is
   obtained by triggering a topology change event and measuring the
   time controller takes to detect and initiate a topology re-discovery
   process.

Measurement Units:
   milliseconds

See Also:
   None
[ACM] - stopped here

_______________________________________________
bmwg mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/bmwg