Opsdir last call review of draft-ietf-bmwg-sdn-controller-benchmark-meth-07

[Scott Bradner <[e-mail]>]

Reviewer: Scott Bradner
Review result: Has Issues

This ID specifies the methodology to be used in testing the performance of SDN
controllers. It is a standard BMWG methodology document and thus, cannot have
any impact, operational or otherwise, on operational networks - see RFC 6815

follow are some comments on the document itself

section 4.1 - Leaf-Spine used but not defined
section 4.4 - using old software versions seems to add complexity with little
benefit section 4.7 - would be useful to say that the variance over the
repeated tests should be reported in the "test reporting" section - e.g., in
section 5.1.1 I would add a row that reports the variance - same for all tests
section 5.1.1 - is the topology discovery process slow enough that a 3 second
granularity of the measurement will show differences between systems? section
5.1.2 - procedure step 2 - what inserts the timestamp in the response message
(R1)? section 5.1.2 - the ID says "successful messages exchanged" - might it be
useful to report the % of unsuccessful exchanges? section 5.1.6 - the title &
objective description do not match - the title says "rate" but the objective is
a count - the test seems to try to get the rate section 5.1.7 - same issue as
section 5.1.6 section 5.2.3 - procedure step 1 - this reads as if the addresses
are unique but unchanging - if that is not what is meant then it should
specifically say that the addresses are changing section 5.3.1 - it might be
useful to establish specific "invalid messages" since I could imagine that the
devices could handle different types in different ways that could have an
impact on speed of handling section 5.3.2 - "a huge number" is somewhat
undefined
        do you mean to say TCP SYN messages rather than TCP SYNC messages?
section 6 - RFC 2026 is referenced in the introduction but not listed in the
references section 9 - I have now retired so no affiliation should be listed