Re: [bmwg] SIP benchmarking drafts
"Banks, Sarah" <sbanks@akamai.com> Tue, 18 February 2014 18:03 UTC
Return-Path: <sbanks@akamai.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7D4011A0504 for <bmwg@ietfa.amsl.com>; Tue, 18 Feb 2014 10:03:13 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.448
X-Spam-Level:
X-Spam-Status: No, score=-2.448 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.548] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 70F8btw4TCQN for <bmwg@ietfa.amsl.com>; Tue, 18 Feb 2014 10:03:04 -0800 (PST)
Received: from prod-mail-xrelay06.akamai.com (prod-mail-xrelay06.akamai.com [96.6.114.98]) by ietfa.amsl.com (Postfix) with ESMTP id 78C5C1A067C for <bmwg@ietf.org>; Tue, 18 Feb 2014 10:03:04 -0800 (PST)
Received: from prod-mail-xrelay06.akamai.com (localhost.localdomain [127.0.0.1]) by postfix.imss70 (Postfix) with ESMTP id 3C6411655DB; Tue, 18 Feb 2014 18:03:01 +0000 (GMT)
Received: from prod-mail-relay03.akamai.com (prod-mail-relay03.akamai.com [172.27.8.26]) by prod-mail-xrelay06.akamai.com (Postfix) with ESMTP id 315B61655CD; Tue, 18 Feb 2014 18:03:01 +0000 (GMT)
Received: from ustx2ex-cashub.dfw01.corp.akamai.com (ustx2ex-cashub7.dfw01.corp.akamai.com [172.27.25.73]) by prod-mail-relay03.akamai.com (Postfix) with ESMTP id 2B8962FD65; Tue, 18 Feb 2014 18:03:01 +0000 (GMT)
Received: from USMBX2.msg.corp.akamai.com ([169.254.1.225]) by ustx2ex-cashub7.dfw01.corp.akamai.com ([172.27.25.73]) with mapi; Tue, 18 Feb 2014 12:03:00 -0600
From: "Banks, Sarah" <sbanks@akamai.com>
To: "Vijay K. Gurbani" <vkg@bell-labs.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Date: Tue, 18 Feb 2014 12:02:57 -0600
Thread-Topic: [bmwg] SIP benchmarking drafts
Thread-Index: Ac8s06jFufX61JWsSDCTSA+iDGxeTg==
Message-ID: <CF28E050.136F%sbanks@akamai.com>
References: <5303757F.90806@bell-labs.com>
In-Reply-To: <5303757F.90806@bell-labs.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.3.9.131030
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: http://mailarchive.ietf.org/arch/msg/bmwg/sOn_p-0EE89JHkE2Q47y0UgJpNM
Cc: "draft-ietf-bmwg-sip-bench-term@tools.ietf.org" <draft-ietf-bmwg-sip-bench-term@tools.ietf.org>, Robert Sparks <rjsparks@nostrum.com>
Subject: Re: [bmwg] SIP benchmarking drafts
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Feb 2014 18:03:13 -0000
Thanks for the update Vijay, it's nice to see. See you in London! /S On 2/18/14, 7:00 AM, "Vijay K. Gurbani" <vkg@bell-labs.com> wrote: >Folks: Carol, Scott and I have submitted the SIP benchmarking drafts >for discussion in London. > >These drafts were in IETF LC [1,2] back in January 2013. As part >of IETF LC, Robert Sparks performed an ind-epth review of the drafts >[3]. The revised version of the drafts are based on attending to >his review. > >We will follow up with a detailed list of changes. Here is a high- >level list of changes outside the editorial changes to improve >readability: > >- Simplified the testing of SIP devices. We no longer maintain > test setup parameters (Terminology, or T) related to forking and > loop detection. While these add time for each SIP transaction > to complete, tests for these would be in essence protocol conformance > tests, not benchmarking tests. > > Similarly, we have removed the test suites related to loop detection > and forking from the Methodology (M) document as well. > >- We have simplified the benchmarks being sought in T and M to three: > Session Establishment Rate, Registration Rate, and Registration > Attempt Rate. Earlier sesison-related benchmarks (Session Capacity, > Session Overload Capacity, Session Establishment Performance, and > Session Attempt Delay) have been taken out for the simple reason that > our implementation experience indicated that these additional > benchmarks do not provide much benefit beyond what is provided by > the singularly important Session Establishment Rate benchmark. > Furthermore, reviewers also had questions on the nature of these > benchmarks and their uniform interpretation. > >- We have taken out benchmark related to IM Rate due to the > variabilities inherent in benchmarking it; for instance, size of > payload, fragmentation potential on large payloads, varying user > behaviour in the real world (an IM may pend until the subscriber > actually reads it and replies), etc. > > The original intent of using IM was to benchmark a simple non-INVITE > transaction. The current version of the draft takes the tact that > the REGISTER transaction better serves this purpose. > >- We have expanded the test reporting template to include artifacts > related to TLS ciphersuites (for TLS-based benchmarks) and IPSec > profiles (for IPSec-based benchmarks). > >In sum, we believe that the reduced focus of the benchmarking results >in a vastly more tractable system whose properties the testing >organizations can understand (and control) much better. The results >from the improved benchmarks will provide an authoritative answer for >comparing different vendor offerings or understanding the behaviour of >the device under test. > >-09 of terminology is available in [4] and methodology is available >in [5]. > >Comments are welcome. > >[1] http://www.ietf.org/mail-archive/web/bmwg/current/msg02717.html >[2] http://www.ietf.org/mail-archive/web/bmwg/current/msg02718.html >[3] http://www.ietf.org/mail-archive/web/bmwg/current/msg02719.html >[4] http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-09 >[5] http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-09 > >Cheers, > >- vijay >-- >Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent >1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA) >Email: vkg@{bell-labs.com,acm.org} / vijay.gurbani@alcatel-lucent.com >Web: http://ect.bell-labs.com/who/vkg/ | Calendar: http://goo.gl/x3Ogq > >_______________________________________________ >bmwg mailing list >bmwg@ietf.org >https://www.ietf.org/mailman/listinfo/bmwg
- Re: [bmwg] SIP benchmarking drafts Scott Poretsky
- Re: [bmwg] SIP benchmarking drafts Banks, Sarah
- [bmwg] SIP benchmarking drafts Vijay K. Gurbani