Re: [bmwg] Fwd: Review: draft-ietf-bmwg-sip-bench-term-10 and draft-ietf-bmwg-sip-bench-meth-10

[Carol Davids <[e-mail]>]

Thank you too for your considerable effort and support!  - Carol and Vijay

From:  Robert Sparks <[e-mail]>
Date:  Thursday, July 3, 2014 at 12:54 PM
To:  Carol Davids <[e-mail]>, BMWG mailing list BMWG mailing list
<[e-mail]>
Subject:  Re: [bmwg] Fwd: Review: draft-ietf-bmwg-sip-bench-term-10 and
draft-ietf-bmwg-sip-bench-meth-10

    
 Thank you for the quick revisions, and the detailed responses below.
 I've reviewed the diffs, and reread the resulting drafts.
 All my concerns have been addressed.
 
 Thanks again for the effort you've put into this!
 
 RjS
 
 
On 7/2/14, 6:13 PM, Carol Davids wrote:
 
 
>             
>  
> Robert,
>  
> 
>  
>  
> Thank you very much for you careful reviews.  Your question and comments have
> greatly helped us to restructure and cleanup the documents. The versions 11 of
> the documents are now available.  A summary of changes is below. Details of
> the resolution of each item in the review are in­line in our reply.
>  
> 
>  
>  
> Summary of changes:
>  
> *Made it explicit that we do not intend for the EA server to add delay before
> it initiates its 200 response. (M)
>  
> *Clarified the need for distinct AoR¹s in the REGISTER tests (M)
>  
> *Clarified SIP Transport Protocol issues (M)
>  
> *Added a section called Connection-oriented Transport Management to address
> the question of the number of such connections to be used. (M)
>  
> *Added rows in the Test Setup Report for Connection-oriented Transport as well
> as for Codec Type (M)
>  
> *Clarified use of the parameter, Œmedia packet size¹ is for audio codecs. (T)
>  
> 
>  
>  
> *Removed references to IS/NS.  (M), (T)
>  
> *Removed references to session(sig,medc,med) and the associated diagram. (T)
>  
> *Added a new figure and explanatory material for registration tests, including
> Œexpected results.' (M)
>  
> *Removed reference to RTSP
>  
> 
>  
>  
> *Cleaned up typos and other nits.
>  
> 
>  
>  
> See detailed resolutions in-line below.
>  
> 
>  
>  
> Best Regards,
>  
> 
>  
>  
> Carol Davids 
>  
> Vijay Gurbani 
>  
> 
>  
>  
>  
> 
>  
>   
> From:  Robert Sparks <[e-mail]>
>  Date:  Wednesday, June 18, 2014 at 4:26 PM
>  To:  BMWG mailing list BMWG mailing list <[e-mail]>
>  Subject:  [bmwg] Fwd: Review: draft-ietf-bmwg-sip-bench-term-10 and
> draft-ietf-bmwg-sip-bench-meth-10
>  
>  
> 
>  
>  
>   
>  Oops - meant to copy the list. Forwarding...
>  
> 
>  
>  -------- Original Message --------
>  Subject:   Review: draft-ietf-bmwg-sip-bench-term-10 and
> draft-ietf-bmwg-sip-bench-meth-10
>  Date:  Wed, 18 Jun 2014 16:19:06 -0500
>  From:  Robert Sparks <[e-mail]> <mailto:[e-mail]>
>  To:  MORTON, ALFRED C (AL) <[e-mail]> <mailto:[e-mail]> ,
> Vijay Gurbani <[e-mail]> <mailto:[e-mail]> , joel jaeggli
> <[e-mail]> <mailto:[e-mail]> , Carol Davids ([e-mail])
> <[e-mail]> <mailto:[e-mail]> , [e-mail]
> <[e-mail]> <mailto:[e-mail]>
>  CC:  Banks, Sarah ([e-mail]) <[e-mail]>
> <mailto:[e-mail]>
>  
>  
> Reviews of draft-ietf-bmwg-sip-bench-term-10 and
> draft-ietf-bmwg-sip-bench-meth-10
> 
> Thank you for the restructure and cleanup work on these drafts.
>  
>  
>  
>   
> 
>  
>  
> Thank you for your help with this.
>   
>  
>  
>  
> Most of the comments I made on version -08 of each of them have been
> addressed.
> Since there were so many changes, rather than revisit the thread from
> the -08 review,
> I'll start a new thread here, bringing up unaddressed points from the
> earlier review
> as necessary.
> 
> Again, thank you for the adjustment to the title and the text to address
> the scope
> of the documents. The result is much more straightforward and clear than
> what I
> had earlier reviewed.
> 
> I have a few remaining points and questions and a few nits to call out.
> 
> Major technical points:
> 
> 1) I've read through this enough times now that maybe I've become blind
> to where
> it's discussed, but for the INVITE tests, I think you have an unstated
> assumption
> that the responding EA is configured to send 200s as quickly as it can,
> and that
> whatever delay it has in responding is fairly constant. Otherwise, your
> tests
> will have widely varying results due to retransmissions of the INVITE.
> Is it your
> intent that the EA would ever retransmit? If not, the assumptions above
> should
> be spelled out in the methodology document.
>  
>  
>  
>   
> 
>  
>  
> Added new text  in Methodology Section 4.9.
>   
>  
>  
>  
> 2) The documents still don't make it clear that each register request needs
> to be to a distinct AOR (otherwise, there is no sense in having a separate
> re-registration test).
>  
>  
>  
>   
> 
>  
>  
> Added text in Methodology Sections 6.7 and 6.8.
>   
>  
>  
>  
> 3) The documents (particularly the report forms) assume you will use the
> same transport on both sides of the DUT. Please state that explicitly, or
> if allowing for (for instance) UDP on one side of a proxy and TCP on the
> other was intended, please adjust the document to talk about it.
>  
>  
>  
>   
> 
>  
>  
> Added text in Methodology Section 5.1 under the SIP Transport Protocol line.
>   
>  
>  
>  
> 4) I think the documents are assuming that an EA will make one connection
> if it's using a connection oriented transport (like TLS). The results of
> the test will be very different if it opens a new connection for each
> sent message. Similarly, I think you're assuming that a connection
> gets set up between the DUT and the EA that will respond to an INVITE
> once. Those should be called out explicitly. If you're not assuming
> that, there needs to be more discussion about how connections get
> established, and whether that's a parameter that needs to be captured
> as part of the test.
>  
>  
>  
>   
> 
>  
>  
> Added a new section, 4.2 in Methodology, called Connection-oriented Transport
> Management.
>  
> 
>  
>  
> Added corresponding line in Methodology, Section 5.1, called Connection
> Management Strategy.
>   
>  
>  
>  
> 5) Is it the intent to only test with media that acts like G.711 over
> RTP? The configuration parameters you have for EA imply that. If its
> not the case, are you missing configuration parameters for, say, video
> codecs (an EA won't be able to use what you have in terminology's 3.3.4
> for that stream), or for an msrp media session?
>  
>  
>  
>   
> 
>  
>  
> Added Section 4.7 in Methodology, entitled ŒCodec Type.¹
>  
> 
>  
>  
> Added text to Section 3.3.4 Definition and Discussion, in Terminology.
>   
>  
>  
>  
> Major editorial points:
> 
> 1) The terms IS and NS are holdovers from when the document was trying to
> do more than it is now. NS is not accurately defined and is at cross
> purposes with the tests you _do_ define in the document (The discussion
> of MESSAGE and SUBSCRIBE that's still in 3.1.1 of the terminology document
> does not help this document at all.) I know these words took effort to
> write,
> but they really should just be removed. A very short introduction to the two
> types of session you are actually testing would leave much less room for
> confusion.
>  
>  
>  
>   
> 
>  
>  
> Removed reference to IS and NS.  Replaced those terms by 'invitation to
> audio/video communication' and Œregistration¹ respectively.
>   
>  
>  
>  
> 2) I challenged the usefulness of the definition of session
> (sig,medc,med) and
> the diagram trying to show these as points in some three-dimensional
> space in
> my previous review.  I did see Carol's explanation of it in the summary
> to that
> review, but I disagree that it is helping this document. I think it's
> hurting
> by adding confusion.  If you deleted it, the rest of the document means
> what
> it meant before, and the reader is no less informed.  Please remove it.
> If the primary point is to make sure that the tester considers covering
> every
> permutation of the test parameters, just say that. The graphic implies that
> some things have more sess.sig than others, and that you might talk about
> the distance between points in this vector space. Save all this aside and
> bring it back in a document that actually uses it if you must, but please
> take it out of this one.
>  
>  
>  
>   
> 
>  
>  
> Agreed.  See changes in Methodology Section 3.1.1.
>  
> 
>  
>   
>  
>  
>  
> 3) The code in Appendix A of the methodology doc should be identified as
> a "Code Component" as described in the TLP
> (http://trustee.ietf.org/license-info/IETF-TLP-4.htm)
> and a license block should be added.
>  
>  
>  
>   
> 
>  
>  
> Added to Appendix A.
>   
>  
>  
>  
> More minor points.
> 
> * The methodology document says the DUT can be any conforming 3261 device.
> The terminology document says the device can't be user equipment (see
> bullet 1
> in section 1.1) . Neither are correct. (A presence server, for instance, is
> not a reasonable thing for a DUT for the tests in this document). I
> suggest in
> both documents you just explicitly list what the DUT can be. That list
> should
> not include "User Agent Server" as it currently does in 3.2.2 of the
> terminology
> document. A UAS is just a role that any of these devices, including
> end-user
> terminals, can hold.
>  
>  
>  
>   
> 
>  
>  
> Agreed. 
>  
> Changes made in Terminology, Section 2.1 bullets 2 and 3; and Section 3.2.2,
> Discussion. 
>  
> Changes made in Methodology, Section 2, the paragraph beginning,"The
> device-under-test (DUT) is.."
>      
>   
>  
>  
>  
> * If a Registrar is the DUT, then neither of the topology figures are
> correct.
> (Unless you're intending for an EA to be the registrar, and the DUT is just
> a proxy). Consider adding a figure that more clearly shows what the topology
> for your registration test is intended to be, and making it clear that you
> are only testing INVITE through devices that forward messages.
>  
>  
>  
>   
> Changes made in Methodology Section 3, Benchmarking Topology: The new figure,
> Figure 3,  and explanatory material were added.
>   
>  
>  
>  
> * There are several places that call out RTSP as a media protocol.
> RTSP is a control protocol, not a media protocol - it doesn't make sense
> being listed with RTP and SRTP. MSRP would make more sense.
>  
>  
>  
>   
> 
>  
>  
> Agreed. Reference to RTSP was removed.
>   
>  
>  
>  
> * The Expected Results in Section 6.8 of the methodology claim that
> the rate should not be more than what was in 6.7. How do you come to
> that conclusion? There are several valid implementation choices that
> could lead to reregistration taking slightly longer than an initial
> registration.
>  
>  
>  
>   
> 
>  
>  
> See changes in Expected Results in Methodology Section 6.8, as well as
> clarification of the procedure and reference to the new Figure 3.
>   
>  
>  
>  
> NITS:
> 
> Methodology document:
> 
> Introduction paragraph 1 s/in Terminology document/in the Terminology
> Document/
>  
>  
>  
>   
> done
>   
>  
>  
>  
> Introduction paragraph 5. This points to section 4 (the IANA
> consideration section)
> and section 2 (the introduction) of the terminology document for an
> explanation
> of configuration options. Neither of those sections explain
> configuration options.
> Where did you mean to point?
>  
>  
>  
>   
> 
>  
>  
> Deleted reference.
>   
>  
>  
>  
> Terminology document:
> 
> Security Considerations: Please remove "and various other drafts". If
> you know of
> other important documents to point to, ase add them as references.
>  
>  
>  
>   
> 
>  
>  
> Removed
>   
>  
>  
>  
> The definition of Stateful Proxy and Stateless Proxy copied the words
> "defined
> by this specification" from RFC3261. This literal copy introduces ambiguity.
> Please replace "by this specification" with "by [RFC3261]".
>  
>  
>  
>   
> 
>  
>  
> Done
>   
>  
>  
>  
> Introduction paragraph 4, last sentence: By calling out devices that include
> the UAS and UAC functions, you have eliminated stateless proxies, which
> contain
> neither.
>  
>  
>  
>   
> 
>  
>  
> We excluded stateless proxies from the scope of the documents, and explained
> that in Section 2,  paragraph 2 (Introduction) to the Methodology document.
>   
>  
>  
>  
> In section 3, when you use the templates, you are careful to say None under
> issues when there are no issues. Please use the same care with See Also.
> Right now, you have empty See Also: sections that could be misread to
> take up whatever content follows (particularly by a text-to-speech engine).
>  
>  
>  
>   
> 
>  
>  
> Added Œnone' as needed.
>   
>  
>  
>  
> 3.1.6: s/tie interval/time interval/
>  
>  
>  
>   
> 
>  
>  
> Done
>   
>  
>  
>  
>  
>  
>  
>  
>  
>  _______________________________________________ bmwg mailing list
> [e-mail] https://www.ietf.org/mailman/listinfo/bmwg
>  
>  
> _______________________________________________ bmwg mailing list
> [e-mail] https://www.ietf.org/mailman/listinfo/bmwg