Re: [bmwg] WGLC on SIP Benchmarking Drafts (04)

[Al Morton <[e-mail]>]

At 02:48 PM 4/19/2012, Al Morton wrote:
>A WG Last Call period for the Internet-Drafts on SIP Device
>Benchmarking:
>
>http://www.ietf.org/id/draft-ietf-bmwg-sip-bench-term-04.txt
>
>http://www.ietf.org/id/draft-ietf-bmwg-sip-bench-meth-04.txt
>
>will be open from 19 April 2012 through 18 May 2012.


Below are the comments I've collected on these versions.
Al
(as a participant)

-=-=-=-=-=-=-=-=-=-=-

There are a few minor inconsistencies between the Abstract and the Scope
in the term draft. Some may be sorted out by using clear terminology.

Abstract (3rd sent.)
...                                      The performance benchmark
    metrics are obtained for the SIP control plane and media plane.
Scope
...
     o  Control Signaling in presence of Media
       *  The media performance is not benchmarked in this work item.

Also,
Abstract
... It is critical to provide test setup parameters and a
    methodology document for SIP performance benchmarking because SIP
    allows a wide range of configuration and operational conditions that
    can influence performance benchmark measurements.
Scope
    The scope of this work item is summarized as follows:
    o  This terminology document describes SIP signaling (control- plane)
       performance benchmarks for black-box measurements of SIP
       networking devices.  Stress and debug scenarios are not addressed
       in this work item.   ^^^^^^
Later in the Scope
    o  SIP Overload [I-D.ietf-soc-overload-design] is within the scope of
       this work item.  We test to failure and then can continue to
       observe and record the behavior of the system after failures are
       recorded.
 >>> Isn't Overload a form of Stress testing?

-=-=-=-=-=-=-=-=-=-=-=-

Section 3.1.1

       Sessions will be represented as a vector array with three
       components, as follows:
       session->
       session[x].sig, the signaling component
       session[x].medc, the media control component (e.g.  RTCP)
       session[x].med[y], an array of associated media streams (e.g.

Question: Why is media control not indicated as a value on the z-axis?
       session[x].medc[z], the media control component (e.g.  RTCP)

If medc is a only a function of session[x], then is the 3rd axis necessary?

-=-=-=-=-=-=-=-=-=-=-

Section 3.1.5 Overload

    Definition:
       Overload is defined as the state where a SIP server does not have
       sufficient resources to process all incoming SIP messages
       [I-D.ietf-soc-overload-design].  The distinction between an
       overload condition and other failure scenarios is outside the
       scope of this document which is blackbox testing.

Does the last sentence really mean this?

Achieving Overload state does not require a level of SIP message load
sufficient to cause failure. Testing with under failure conditions is
beyond the scope of this document.

And, it would be good to say whether Overload state can be considered
part of Stress conditions (or remove the word Stress from the Scope?).


Also:

    Issues:
       The issue of overload in SIP networks is currently a topic of
       discussion in the SIPPING WG.  The normal response to an overload
       stimulus -- sending a 503 response -- is considered inadequate and
       new response codes and behaviors may be specified in the future.
       From the perspective of this document, all these responses will be
       considered to be failures.  There is thus no dependency between
       this document and the ongoing work on the treatment of overload
       failure.

There has been some progress on the overload topic, and an RFC published,
so this section may need to be updated. In general, IETF considers
working groups to be ephemeral, and therefore not mentioned by name
in RFCs.

-=-=-=-=-=-=-=-=-=-=-

3.1.7.  Established Session

    Definition:
       A SIP session for which the EA acting as the UE/UA has received a
       200 OK message from the DUT/SUT.

As a practical matter, how long does one wait for 200 OK?
Is there a standard time-out to cite here?

-=-=-=-=-=-=-=-=-=-=-

3.1.9.  Non-INVITE-initiated Session (NS)

    Definition:
       A session that is created by an exchange of messages in the
       Signaling Plane that does not include an initial SIP INVITE
       message.

s/an exchange/any exchange/  ??

-=-=-=-=-=-=-=-=-=-=-=-


3.4.4.  Session Overload Capacity

    Definition:
       The maximum number of Established Sessions that can exist
       simultaneously on the DUT/SUT until it stops responding to Session
       Attempts.

Again, the waiting time for 200 OK figures in this benchmark,
and any other that relies on the Established Session definition.

-=-=-=-=-=-=-=-=-=-=-=-=-

5.  Security Considerations

    Documents of this type do not directly affect the security of
    Internet or corporate networks as long as benchmarking is not
    performed on devices or systems connected to production networks.
    Security threats and how to counter these in SIP and the media layer
    is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711
    [RFC3711] and various other drafts.  This document attempts to
    formalize a set of common terminology for benchmarking SIP networks.
add ... constructed in isolated test environments (not production networks).

(same comment for methodology draft)

These considerations below seem to apply to another draft, remove?

    Packets with unintended and/or unauthorized DSCP or IP precedence
    values may present security issues.  Determining the security
    consequences of such packets is out of scope for this document.