Re: [bmwg] {Spam?} RE: Opsdir last call review of draft-ietf-bmwg-sdn-controller-benchmark-meth-07

[[e-mail]]

Thanks Al for formatting Scott's comment.
Hi Scott,
Thank you for reviewing the draft and providing your comments. Please find below our response inline to your comments. 
Best Regards,
Bhuvan

On Tue, Jan 30, 2018 at 08:02 PM, "MORTON, ALFRED C (AL)"  wrote:
Hi Scott,
Thanks for your review and insightful comments!

Something modified your original formatting, so
I've tried to restore it below.  Authors, please 
reply to Scott's comments and add changes to your 
working version of the draft.

regards,
Al
doc shepherd

-=-=-=-=-=-=-=-=-=-=-=-

Reviewer: Scott Bradner
Review result: Has Issues

This ID specifies the methodology to be used in testing the performance of SDN
controllers. It is a standard BMWG methodology document and thus, cannot have
any impact, operational or otherwise, on operational networks - see RFC 6815

The following  are some comments on the document itself:

section 4.1 - Leaf-Spine used but not defined
[Bhuvan] We will add the definition in the terminology document 

section 4.4 - using old software versions seems to add complexity with little
benefit 
[Bhuvan] We agree with your observation. We have added this verification to ensure backward compatibility. In fact we have given this example (older vs current) for better clarity. 

section 4.7 - would be useful to say that the variance over the
repeated tests should be reported in the "test reporting" section 
- e.g., in section 5.1.1 I would add a row that reports the variance 
- same for all tests
[Bhuvan] We will add a row to report variance across all tests 

section 5.1.1 - is the topology discovery process slow enough that a 3 second
granularity of the measurement will show differences between systems? 
[Bhuvan] We have considered 3 seconds for the worst case scenario 

section 5.1.2 - procedure step 2 - what inserts the timestamp in the response message
(R1)? 
[Bhuvan] The intent was to find the difference between the transmitted time of request and the received time of the corresponding response. I think we need to rephrase the text as follows.  '2. Record every request transmit (T1) time and the corresponding response (R1) received time at the forwarding plane test emulator interface (I1) for every successful message exchange' 

section 5.1.2 - the ID says "successful messages exchanged" - might it be
useful to report the % of unsuccessful exchanges? [Bhuvan] We will report the % of unsuccessful exchanges as well 
section 5.1.6 - the title & objective description do not match 
- the title says "rate" but the objective is a count 
- the test seems to try to get the rate section 5.1.7 
- same issue as section 5.1.6 [Bhuvan] We will reword the objective section to reflect the rate. 
section 5.2.3 - procedure step 1 
- this reads as if the addresses are unique but unchanging 
- if that is not what is meant then it should specifically say that 
the addresses are changing [Bhuvan] The intent is to generate unique src/dst address combinations to populate the forwarding table and measure its capacity. Could you please clarify what you mean by unchanging in Step 1. ?
section 5.3.1 - it might be useful to establish specific "invalid messages" 
since I could imagine that the devices could handle different types in 
different ways that could have an impact on speed of handling 
[Bhuvan] We have clarified the invalid message in the Note section. Do we need to add the info in the test report as well? 

section 5.3.2 - "a huge number" is somewhat undefined
do you mean to say TCP SYN messages rather than TCP SYNC messages?
[Bhuvan] Yes. We will address this comment in the revised draft. 

section 6 - RFC 2026 is referenced in the introduction but not listed in the
references 
[Bhuvan] We will include RFC 2026 in the reference  

section 9 - I have now retired so no affiliation should be listed
[Bhuvan] We will remove the affiliation 

-----Original Message-----
From: bmwg [mailto:[e-mail] (mailto:[e-mail])] On Behalf Of Scott Bradner
Sent: Monday, January 29, 2018 9:17 PM
To: [e-mail] (mailto:[e-mail])
Cc: [e-mail] (mailto:[e-mail]); [e-mail] (mailto:[e-mail]); draft-ietf-bmwg-sdn-controller-
[e-mail] (mailto:[e-mail])
Subject: [bmwg] Opsdir last call review of draft-ietf-bmwg-sdn-
controller-benchmark-meth-07

Reviewer: Scott Bradner
Review result: Has Issues

This ID specifies the methodology to be used in testing the performance
of SDN
controllers. It is a standard BMWG methodology document and thus, cannot
have
any impact, operational or otherwise, on operational networks - see RFC
6815

follow are some comments on the document itself

section 4.1 - Leaf-Spine used but not defined
section 4.4 - using old software versions seems to add complexity with
little
benefit section 4.7 - would be useful to say that the variance over the
repeated tests should be reported in the "test reporting" section -
e.g., in
section 5.1.1 I would add a row that reports the variance - same for all
tests
section 5.1.1 - is the topology discovery process slow enough that a 3
second
granularity of the measurement will show differences between systems?
section
5.1.2 - procedure step 2 - what inserts the timestamp in the response
message
(R1)? section 5.1.2 - the ID says "successful messages exchanged" -
might it be
useful to report the % of unsuccessful exchanges? section 5.1.6 - the
title &
objective description do not match - the title says "rate" but the
objective is
a count - the test seems to try to get the rate section 5.1.7 - same
issue as
section 5.1.6 section 5.2.3 - procedure step 1 - this reads as if the
addresses
are unique but unchanging - if that is not what is meant then it should
specifically say that the addresses are changing section 5.3.1 - it
might be
useful to establish specific "invalid messages" since I could imagine
that the
devices could handle different types in different ways that could have
an
impact on speed of handling section 5.3.2 - "a huge number" is somewhat
undefined
do you mean to say TCP SYN messages rather than TCP SYNC
messages?
section 6 - RFC 2026 is referenced in the introduction but not listed in
the
references section 9 - I have now retired so no affiliation should be
listed

_______________________________________________
bmwg mailing list
[e-mail] (mailto:[e-mail])
https://urldefense.proofpoint.com/v2/url?u=https (https://urldefense.proofpoint.com/v2/url?u=https)-
3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Oip5-
2t1wtJ1ZpUIlQWQcvgNAVgydAS4jCoQi_LL50Q&s=WP7X4VUxjR59g9gvjUFj_8zTQrz-
iKc2jY9FOaPTlZY&e=

DISCLAIMER: Privileged and/or Confidential information may be
contained in this message. If you are not the addressee of this message,
you may not copy, use or deliver this message to anyone. In such
event,you should destroy the message and kindly notify the sender by
reply e-mail.
It is understood that opinions or conclusions that do not relate to the
official business of the company are neither given nor endorsed by the
company.