Re: [bmwg] Genart last call review of draft-ietf-bmwg-sdn-controller-benchmark-meth-07

Hi Stewart Bryant,
Thank you for reviewing the draft and sharing your comments.
Please find below our responses inline

Thanks,
Bhuvan
On Tue, Jan 30, 2018 at 10:54 PM, Stewart Bryant  wrote:
Reviewer: Stewart Bryant
Review result: Ready with Nits

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair.  Please treat these comments just
like any other last call comments.

For more information, please see the FAQ at

.

Document: draft-ietf-bmwg-sdn-controller-benchmark-meth-07
Reviewer: Stewart Bryant
Review Date: 2018-01-30
IETF LC End Date: 2018-02-02
IESG Telechat date: Not scheduled for a telechat

Summary:

This is a well written comprehensive test set for SDN controllers. It could be
published as is, but some thought about how to address the issues below might
be helpful to the user of this technology. Major issues: None

Minor issues:

I find the large amount of text on Openflow that appears out of the blue in the
appendix somewhat strange. The test suit is controller protocol agnostic, so I
wonder why so much text is devoted to this specific SDN control protocol. If
they are there by way of illustrative example of packet exchanges, it might be
useful to the reader to point to them from the measurement text.
[Bhuvan] We agree with you. But we wanted to explain this methodology using some well established protocol for better understanding. Since OpenFlow is widely used , we have given illustration of this methodology using OF protocol 

Something I am slightly surprised by is the lack of statistical sophistication.
Average is a very crude metric giving no information on the distribution of the
results.
[Bhuvan] We are planning to add variance to all tests besides average metric. Hope this address your concern. 

I imagine that it is now ingrained in this aspect of the industry to specify
graphs and tables, but I would have expected that the results would be
specified in some machine readable format such as xml for input to a database
rather than in the human readable format that is hard coded into this
specification.
[Bhuvan] We are planning to leave this option to individuals performing the methodology.
Nits/editorial comments:

Abstract

This document defines the methodologies for benchmarking control
plane performance of SDN controllers. Terminology related to
benchmarking SDN controllers is described in the companion
terminology document.

SB> It would be convenient to the reader to provide the reference to or name of
SB> the companion document - the twin of the comment in the other review.

SB> it would also be useful to include such a reference early in the main text.
[Bhuvan] We will specify the name of the companion document explicitly 

=============

4. Test Considerations

4.1. Network Topology

The test cases SHOULD use Leaf-Spine topology with at least 1
Network Device in the topology for benchmarking.
SB> Leaf-Spine could use a reference. In Fig 2 I am not sure this is SL rather
than SB> a linear sequence of nodes. There is a better SL diagram later in the
SB> document and it would be useful to the reader to forward reference it.[Bhuvan]  We will work on the figures to reflect Leaf-Spine topo 
========

The test traffic
generators TP1 and TP2 SHOULD be connected to the first and the last
leaf Network Device.

SB> I am sure I know what does first and last mean, but the meaning should be
called out.
[Bhuvan] I agree with you. We mean TP1 should be connected to the flow source endpoint and TP2 should be connected to the flow destination endpoint. 
=========

Procedure:

5. Stop the trial when the discovered topology information matches
the deployed network topology, or when the discovered topology
information return the same details for 3 consecutive queries.

SB> What do you report in the latter case?
[Bhuvan] Step 5 is just a test stop criteria. Both cases we report topology discovery time (as in Step 6)
===========

_______________________________________________
bmwg mailing list
bmwg@ietf.org (mailto:bmwg@ietf.org)
https://www.ietf.org/mailman/listinfo/bmwg (https://www.ietf.org/mailman/listinfo/bmwg)

DISCLAIMER: Privileged and/or Confidential information may be
contained in this message. If you are not the addressee of this message,
you may not copy, use or deliver this message to anyone. In such
event,you should destroy the message and kindly notify the sender by
reply e-mail.
It is understood that opinions or conclusions that do not relate to the
official business of the company are neither given nor endorsed by the
company.