Document: draft-ietf-bmwg-sdn-controller-benchmark-term-09 Reviewer: Stewart Bryant Review Date: 2018-04-16 IETF LC End Date: 2018-02-02 IESG Telechat date: 2018-04-19 Summary: Generally a well written document. The various comments I make are mostly editorial with some on the fringe of being technical. I do not seem to have received a response to a number of issues raised, and as far as I can see there is no change to the text in the issues noted below. Apologies if I have missed your email. Major issues: None Minor issues: I do not seem to have received a response to this issue, and as far as I can see there is no change to the text. Apologies if I have missed your email. *rate* implies actions per unit time, but this does not seem to feature in your definition. 2.3.1.3. Asynchronous Message Processing Rate Definition: The number responses to asynchronous messages (such as new flow SB> That should be the number of responses per second. [Authors] Updated the text to - 'The number responses to asynchronous messages per second..' Discussion: As SDN assures flexible network and agile provisioning, it is important to measure how many network events the controller can handle at a time. This benchmark is obtained by sending asynchronous messages from every connected Network Device at the rate that the controller processes (without dropping them). SB> So what you are testing here is the control network and the SB> controller. This is perhaps the only practical way to run the SB> test, but it seems a pity that you do not deconvolve these SB> two aspects of the test. SB> SB> I suppose this is really network Async Msg Proc rate rather than SB> controller Async proc rate. SB> SB> We may get to this in the companion document, but doesn't there SB> need to be some standardization of the event message to compare SB> apple with apples over time? [Authors] This test benchmarks the controller ability for handling the network events. So it would be difficult to decouple the network & controller aspects. As you suggested, we have provided some standard network events that to be considered for this benchmarking test. Nits/editorial comments: 2.2.4. Number of Cluster nodes Discussion: This parameter is relevant when testing the controller performance in clustering/teaming mode. The number of nodes in the cluster MUST be greater than 1. SB> I see what you are saying, but you may wish to clarify that this SB> constraint does not apply all the time. For example one of two nodes SB> may start later than another, or fail, or maybe I worry over nothing here. [Authors] We have reworded the text that the number of nodes configured in the cluster MUST be greater than 1 and alteast 1 node in the cluster is operationally up. 2.3.2.1. Control Sessions Capacity Measurement Units: SB> Surely this should be in units of sessions? [Authors] Added measurment units - 'Maximum number of sessions' 2.3.2.2. Network Discovery Size Measurement Units: N/A SB> How can this be N/A surely it is a number of network units of various type. [Authors] Added measurment units - 'Maximum number of network nodes and links' 2.3.2.3. Forwarding Table Capacity ------------------------------------------------------------------------------------------------------------------- Spencer Dawkins Spencer Dawkins' No Objection on draft-ietf-bmwg-sdn-controller-benchmark-term-09: (with COMMENT) A nit: The terms defined in this section are extensions to the terms defined in [RFC7426] "Software-Defined Networking (SDN): Layers and Architecture Terminology". This RFC should be referred before attempting to make use of this document. When this draft is published, "this RFC" won't be as clear as it is now (the phrase would also apply to the current document, which would be an RFC). Perhaps "That RFC", or even "RFC 7426" would be clearer. [Authors] Updated the text to 'That RFC' in the revised version There are a lot of measures that say Measurement Units: N/A You might mean "not milliseconds, or some measure like that", but I found it confusing that something like "Trial Repetition" doesn't have measurement units. Saying something like "Number of trials", or even "Integer" would be clearer to me. [Authors] Added Measurement units for Section 2.2. ------------------------------------------------------------------------------------------------------------------- Martin Vigoureux's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-term-09: (with COMMENT) Hello, I wonder about the use of the term "standard" in the abstract in view of the intended status of the document (Informational). Could the use of this word confuse the reader? [Authors] Rephrased the text to 'These two documents provide a method to measure and evaluate the performance of various controller implementations' Also, in the Introduction the word "standard" is used. I don't have the same concern here but wonder if a reference to these standard interfaces shouldn't be provided. [Authors] Rephrased the text to - 'an SDN controller provides the flexibility to program, control, and manage network behaviour dynamically through northbound and southbound interfaces.' Found a few nits found here and there: s/an Network Device/a Network Device/ s/In order to for the controller to/In order for the controller to/ s/This benchmark determine /This benchmark determines/ s/at its Southbound interface ./at its Southbound interface./ [Authors] Fixed all the Nits in the revised version ------------------------------------------------------------------------------------------------------------------- Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-term-09: (with COMMENT) Section 2.3.1.3 [...] This benchmark is obtained by sending asynchronous messages from every connected Network Device at the rate that the controller processes (without dropping them). "obtained" doesn't feel like the right word. [Authors] We updated the text from 'obtained' to 'measured' I'm also a little surprised that there is not consideration to a more-general "acceptable loss fraction" for which the processing rate is determined -- the zero-loss case is certainly interesting, but sometimes it is also useful to know how the system's behavior degrades. [Authors] We begin the first trial by sending asynchronous messages to the controller(s) at the maximum possible rate and re-iterating it until the sdn controller experiences zero loss. We record the message reply rate and the message loss rate which explains the system behaviour degradation. ------------------------------------------------------------------------------------------------------------------- Adam Roach's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-term-09: (with COMMENT) I share Martin's concerns about the use of the word "standard" in this document's abstract and introduction. [Authors] Rephrased the abstract text to 'These two documents provide a method to measure and evaluate the performance of various controller implementations' Also Rephrased the introduction text to - 'an SDN controller provides the flexibility to program, control, and manage network behaviour dynamically through northbound and southbound interfaces.'