Re: [bmwg] Opsdir last call review of draft-ietf-bmwg-sdn-controller-benchmark-meth-07

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Scott,
Thanks for your review and insightful comments!

Something modified your original formatting, so
I've tried to restore it below.  Authors, please 
reply to Scott's comments and add changes to your 
working version of the draft.

regards,
Al
doc shepherd

-=-=-=-=-=-=-=-=-=-=-=-

Reviewer: Scott Bradner
Review result: Has Issues

This ID specifies the methodology to be used in testing the performance of SDN
controllers. It is a standard BMWG methodology document and thus, cannot have
any impact, operational or otherwise, on operational networks - see RFC 6815

The following  are some comments on the document itself:

section 4.1 - Leaf-Spine used but not defined

section 4.4 - using old software versions seems to add complexity with little
benefit 

section 4.7 - would be useful to say that the variance over the
repeated tests should be reported in the "test reporting" section 
 - e.g., in section 5.1.1 I would add a row that reports the variance 
 - same for all tests

section 5.1.1 - is the topology discovery process slow enough that a 3 second
granularity of the measurement will show differences between systems? 

section 5.1.2 - procedure step 2 - what inserts the timestamp in the response message
(R1)? 

section 5.1.2 - the ID says "successful messages exchanged" - might it be
useful to report the % of unsuccessful exchanges? 

section 5.1.6 - the title & objective description do not match 
  - the title says "rate" but the objective is a count 
  - the test seems to try to get the rate section 5.1.7 
  - same issue as section 5.1.6 

section 5.2.3 - procedure step 1 
 - this reads as if the addresses are unique but unchanging 
 - if that is not what is meant then it should specifically say that 
   the addresses are changing 

section 5.3.1 - it might be useful to establish specific "invalid messages" 
since I could imagine that the devices could handle different types in 
different ways that could have an impact on speed of handling 

section 5.3.2 - "a huge number" is somewhat undefined
   do you mean to say TCP SYN messages rather than TCP SYNC messages?

section 6 - RFC 2026 is referenced in the introduction but not listed in the
references 

section 9 - I have now retired so no affiliation should be listed


> -----Original Message-----
> From: bmwg [mailto:[e-mail]] On Behalf Of Scott Bradner
> Sent: Monday, January 29, 2018 9:17 PM
> To: [e-mail]
> Cc: [e-mail]; [e-mail]; draft-ietf-bmwg-sdn-controller-
> [e-mail]
> Subject: [bmwg] Opsdir last call review of draft-ietf-bmwg-sdn-
> controller-benchmark-meth-07
> 
> Reviewer: Scott Bradner
> Review result: Has Issues
> 
> This ID specifies the methodology to be used in testing the performance
> of SDN
> controllers. It is a standard BMWG methodology document and thus, cannot
> have
> any impact, operational or otherwise, on operational networks - see RFC
> 6815
> 
> follow are some comments on the document itself
> 
> section 4.1 - Leaf-Spine used but not defined
> section 4.4 - using old software versions seems to add complexity with
> little
> benefit section 4.7 - would be useful to say that the variance over the
> repeated tests should be reported in the "test reporting" section -
> e.g., in
> section 5.1.1 I would add a row that reports the variance - same for all
> tests
> section 5.1.1 - is the topology discovery process slow enough that a 3
> second
> granularity of the measurement will show differences between systems?
> section
> 5.1.2 - procedure step 2 - what inserts the timestamp in the response
> message
> (R1)? section 5.1.2 - the ID says "successful messages exchanged" -
> might it be
> useful to report the % of unsuccessful exchanges? section 5.1.6 - the
> title &
> objective description do not match - the title says "rate" but the
> objective is
> a count - the test seems to try to get the rate section 5.1.7 - same
> issue as
> section 5.1.6 section 5.2.3 - procedure step 1 - this reads as if the
> addresses
> are unique but unchanging - if that is not what is meant then it should
> specifically say that the addresses are changing section 5.3.1 - it
> might be
> useful to establish specific "invalid messages" since I could imagine
> that the
> devices could handle different types in different ways that could have
> an
> impact on speed of handling section 5.3.2 - "a huge number" is somewhat
> undefined
>         do you mean to say TCP SYN messages rather than TCP SYNC
> messages?
> section 6 - RFC 2026 is referenced in the introduction but not listed in
> the
> references section 9 - I have now retired so no affiliation should be
> listed
> 
> 
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Oip5-
> 2t1wtJ1ZpUIlQWQcvgNAVgydAS4jCoQi_LL50Q&s=WP7X4VUxjR59g9gvjUFj_8zTQrz-
> iKc2jY9FOaPTlZY&e=