[bmwg] Comments on sec 2.3.1 of draft-ietf-bmwg-sdn-controller-benchmark-term-01
"MORTON, ALFRED C (AL)" <acmorton@att.com> Fri, 24 June 2016 23:06 UTC
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 801BA12D76E for <bmwg@ietfa.amsl.com>; Fri, 24 Jun 2016 16:06:48 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.627
X-Spam-Level:
X-Spam-Status: No, score=-5.627 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-1.426, SPF_HELO_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PiHWugD4gvjw for <bmwg@ietfa.amsl.com>; Fri, 24 Jun 2016 16:06:46 -0700 (PDT)
Received: from mail-pink.research.att.com (mail-pink.research.att.com [204.178.8.22]) by ietfa.amsl.com (Postfix) with ESMTP id 7EAD312D1DE for <bmwg@ietf.org>; Fri, 24 Jun 2016 16:06:45 -0700 (PDT)
Received: from mail-green.research.att.com (H-135-207-255-15.research.att.com [135.207.255.15]) by mail-pink.research.att.com (Postfix) with ESMTP id AA112120EF5 for <bmwg@ietf.org>; Fri, 24 Jun 2016 19:16:41 -0400 (EDT)
Received: from exchange.research.att.com (njmtcas1.research.att.com [135.207.255.99]) by mail-green.research.att.com (Postfix) with ESMTP id 2839FE28E1 for <bmwg@ietf.org>; Fri, 24 Jun 2016 19:05:15 -0400 (EDT)
Received: from NJFPSRVEXG0.research.att.com ([fe80::108a:1006:9f54:fd90]) by njmtcas1.research.att.com ([fe80::f1f7:6c06:d0d0:d48c%10]) with mapi; Fri, 24 Jun 2016 19:06:41 -0400
From: "MORTON, ALFRED C (AL)" <acmorton@att.com>
To: "bmwg@ietf.org" <bmwg@ietf.org>
Date: Fri, 24 Jun 2016 19:06:40 -0400
Thread-Topic: Comments on sec 2.3.1 of draft-ietf-bmwg-sdn-controller-benchmark-term-01
Thread-Index: AdHObLepc14ANKfARSuMzh/oHa23hg==
Message-ID: <4AF73AA205019A4C8A1DDD32C034631D4593BBF131@NJFPSRVEXG0.research.att.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/sPTkF4kVFd6PVDSvibKm5DXsJE4>
Subject: [bmwg] Comments on sec 2.3.1 of draft-ietf-bmwg-sdn-controller-benchmark-term-01
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 24 Jun 2016 23:06:48 -0000
Authors, I have several comments and suggestions on 2.3.1, so I send them to you without waiting to go further into the Scalability section. See [ACM] below. Al (as a participant) 2.3. Benchmarking Terms This section defines metrics for benchmarking the SDN controller. The procedure to perform the defined metrics is defined in the accompanying methodology document. 2.3.1. Performance 2.3.1.1. Network Topology Discovery Time Definition: To measure the time taken to discover the network topology - nodes and links by a controller. [ACM] The time taken by a controller to determine the complete network topology, defined as the interval starting with the first discovery message from the controller(s) at its Southbound interface, ending with all features of the static topology determined (and expressed in some form in the controller(s)). Discussion: Network topology discovery is key for the SDN controller to provision and manage the network. So it is important to measure how quickly the controller discovers the topology to learn the current network state. This benchmark is obtained by presenting a network topology (Tree, Mesh or Linear) with the given number of nodes to the controller and wait for the discovery process to complete .It is expected that the controller supports network discovery mechanism and uses protocol messages for its discovery process. [ACM] for metrics like this, we need to make the input parameters clear: Topology: Leaf-Spine, or linear, Topology Size: (e.g., number of leaf switches, number of spine switches) Bhuvan, et al Expires September 19, 2016 [Page 9] Internet-Draft SDN Controller Benchmarking Terminology March 2016 Measurement Units: milliseconds See Also: None 2.3.1.2. Asynchronous Message Processing Time Definition: To measure the time taken by the controller to process an asynchronous message. [ACM] need to anchor the beginning and end of the interval (as above). suggestions? Discussion: For SDN to support dynamic network provisioning, it is important to measure how quickly the controller responds to an event triggered from the network. The event could be any notification messages generated by an Network Device upon arrival of a new flow, link down etc. This benchmark is obtained by sending asynchronous messages from every connected Network Devices one at a time for the defined test duration. This test assumes that the controller will respond to the received asynchronous message. [ACM] regarding the assumption above: When operating at the limits of controller capacity, response messages may be extremely delayed or lost. (IOW, the assumption doesn't hold) Measurement Units: milliseconds See Also: None 2.3.1.3. Asynchronous Message Processing Rate Definition: To measure the maximum number of asynchronous messages that a controller can process within the test duration. Discussion: As SDN assures flexible network and agile provisioning, it is important to measure how many network events that the controller can handle at a time. This benchmark is obtained by sending asynchronous messages from every connected Network Devices at full connection capacity for the given test duration. This test assumes that the controller will respond to all the received asynchronous messages. [ACM] regarding the assumption above: When operating at the limits of controller capacity, response messages may be extremely delayed or lost. (IOW, the assumption doesn't hold) Measurement Units: Messages processed per second. Bhuvan, et al Expires September 19, 2016 [Page 10] Internet-Draft SDN Controller Benchmarking Terminology March 2016 See Also: None 2.3.1.4. Reactive Path Provisioning Time Definition: The time taken by the controller to setup a path reactively between source and destination node, expressed in milliseconds. Discussion: As SDN supports agile provisioning, it is important to measure how fast that the controller provisions an end-to-end flow in the dataplane. The benchmark is obtained by sending traffic from a source endpoint to the destination endpoint, finding the time difference between the first and the last flow provisioning message exchanged between the controller and the Network Devices for the traffic path. [ACM] Here we need input parameters, such as number of nodes in the path, background load on the controller and nodes/switches. THere's also a comparable dataplane measurement, too (First Packet Latency) Measurement Units: milliseconds. See Also: None 2.3.1.5. Proactive Path Provisioning Time Definition: The time taken by the controller to setup a path proactively between source and destination node, expressed in milliseconds. Discussion: For SDN to support pre-provisioning of traffic path from application, it is important to measure how fast that the controller provisions an end-to-end flow in the dataplane. The benchmark is obtained by provisioning a flow on controller's northbound interface for the traffic to reach from a source to a destination endpoint, finding the time difference between the first and the last flow provisioning message exchanged between the controller and the Network Devices for the traffic path. Measurement Units: milliseconds. See Also: None Bhuvan, et al Expires September 19, 2016 [Page 11] Internet-Draft SDN Controller Benchmarking Terminology March 2016 2.3.1.6. Reactive Path Provisioning Rate Definition: Measure the maximum number of independent paths a controller can concurrently establish between source and destination nodes reactively within the test duration, expressed in paths per second. Discussion: For SDN to support agile traffic forwarding, it is important to measure how many end-to-end flows that the controller could setup in the dataplane. This benchmark is obtained by sending traffic each with unique source and destination pairs from the source Network Device and determine the number of frames received at the destination Network Device. [ACM] Here we need input parameters, such as number of nodes in the path, background load on the controller and nodes/switches. Also, this measurement seems to make the assumption that the controller does not loose or discard requests, which may not hold when approaching the Max Rate. Measurement Units: Paths provisioned per second. See Also: None 2.3.1.7. Proactive Path Provisioning Rate Definition: Measure the maximum number of independent paths a controller can concurrently establish between source and destination nodes proactively within the test duration, expressed in paths per second. Discussion: For SDN to support pre-provisioning of traffic path for a larger network from the application, it is important to measure how many end-to-end flows that the controller could setup in the dataplane. This benchmark is obtained by sending traffic each with unique source and destination pairs from the source Network Device. Program the flows on controller's northbound interface for traffic to reach from each of the unique source and destination pairs and determine the number of frames received at the destination Network Device. [ACM] Here we need input parameters, such as number of nodes in the path, background load on the controller and nodes/switches. Also, this measurement seems to make the assumption that the controller does not loose or discard requests, which may not hold when approaching the Max Rate. Measurement Units: Paths provisioned per second. See Also: None 2.3.1.8. Network Topology Change Detection Time Definition: The amount of time required for the controller to detect any changes in the network topology. [ACM] The "any" in this definition is cause for concern, Need some Input parameters, such as: Number or changes, categorized by number of links removed, number os switches/nodes remove. Discussion: In order to for the controller to support fast network failure recovery, it is critical to measure how fast the controller is able to detect any network-state change events. This benchmark is obtained by triggering a topology change event and measuring the time controller takes to detect and initiate a topology re-discovery process. Measurement Units: milliseconds See Also: None [ACM] - stopped here
- Re: [bmwg] Comments on sec 2.3.1 of draft-ietf-bm… Bhuvan (Veryx)
- [bmwg] Comments on sec 2.3.1 of draft-ietf-bmwg-s… MORTON, ALFRED C (AL)