[bmwg] SIP benchmarking drafts

["Vijay K. Gurbani" <[e-mail]>]

Folks: Carol, Scott and I have submitted the SIP benchmarking drafts
for discussion in London.

These drafts were in IETF LC [1,2] back in January 2013.  As part
of IETF LC, Robert Sparks performed an ind-epth review of the drafts
[3].  The revised version of the drafts are based on attending to
his review.

We will follow up with a detailed list of changes.  Here is a high-
level list of changes outside the editorial changes to improve
readability:

- Simplified the testing of SIP devices.  We no longer maintain
   test setup parameters (Terminology, or T) related to forking and
   loop detection.  While these add time for each SIP transaction
   to complete, tests for these would be in essence protocol conformance
   tests, not benchmarking tests.

   Similarly, we have removed the test suites related to loop detection
   and forking from the Methodology (M) document as well.

- We have simplified the benchmarks being sought in T and M to three:
   Session Establishment Rate, Registration Rate, and Registration
   Attempt Rate.  Earlier sesison-related benchmarks (Session Capacity,
   Session Overload Capacity, Session Establishment Performance, and
   Session Attempt Delay) have been taken out for the simple reason that
   our implementation experience indicated that these additional
   benchmarks do not provide much benefit beyond what is provided by
   the singularly important Session Establishment Rate benchmark.
   Furthermore, reviewers also had questions on the nature of these
   benchmarks and their uniform interpretation.

- We have taken out benchmark related to IM Rate due to the
   variabilities inherent in benchmarking it; for instance, size of
   payload, fragmentation potential on large payloads, varying user
   behaviour in the real world (an IM may pend until the subscriber
   actually reads it and replies), etc.

   The original intent of using IM was to benchmark a simple non-INVITE
   transaction.  The current version of the draft takes the tact that
   the REGISTER transaction better serves this purpose.

- We have expanded the test reporting template to include artifacts
   related to TLS ciphersuites (for TLS-based benchmarks) and IPSec
   profiles (for IPSec-based benchmarks).

In sum, we believe that the reduced focus of the benchmarking results
in a vastly more tractable system whose properties the testing
organizations can understand (and control) much better.  The results
from the improved benchmarks will provide an authoritative answer for
comparing different vendor offerings or understanding the behaviour of
the device under test.

-09 of terminology is available in [4] and methodology is available
in [5].

Comments are welcome.

[1] http://www.ietf.org/mail-archive/web/bmwg/current/msg02717.html
[2] http://www.ietf.org/mail-archive/web/bmwg/current/msg02718.html
[3] http://www.ietf.org/mail-archive/web/bmwg/current/msg02719.html
[4] http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-09
[5] http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-09

Cheers,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA)
Email: vkg@{bell-labs.com,acm.org} / [e-mail]
Web: http://ect.bell-labs.com/who/vkg/  | Calendar: http://goo.gl/x3Ogq