Re: [bmwg] My comments on draft-ietf-bmwg-sip-bench-meth-09
Carol Davids <davids@iit.edu> Mon, 14 April 2014 21:23 UTC
Return-Path: <davids@iit.edu>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C148D1A06E6 for <bmwg@ietfa.amsl.com>; Mon, 14 Apr 2014 14:23:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 2.399
X-Spam-Level: **
X-Spam-Status: No, score=2.399 tagged_above=-999 required=5 tests=[BAYES_50=0.8, MANGLED_NAIL=2.3, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jNKSvY4eWDfQ for <bmwg@ietfa.amsl.com>; Mon, 14 Apr 2014 14:23:08 -0700 (PDT)
Received: from mail-ie0-f171.google.com (mail-ie0-f171.google.com [209.85.223.171]) by ietfa.amsl.com (Postfix) with ESMTP id 201301A06AC for <bmwg@ietf.org>; Mon, 14 Apr 2014 14:23:08 -0700 (PDT)
Received: by mail-ie0-f171.google.com with SMTP id ar20so8749424iec.2 for <bmwg@ietf.org>; Mon, 14 Apr 2014 14:23:05 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:user-agent:date:subject:from:to:message-id :thread-topic:references:in-reply-to:mime-version:content-type :content-transfer-encoding; bh=y/M1r+VjABWiAKoBNIU3wVm7uvmSDNSLTWnr4UbJSms=; b=AlSFwLuBprpvBniwLwiKBiQx6tBOZjns3T3jiipIVdFJnCnvEEP5nYCrlYYy6EMWB+ EdUEiSTexg7iAGGw6z3ybL73VttIPzHEyU7XV8UsGHFozsso4PJi4/YFmlyQyDu35SXU 07CkxLFoIfm4Kjp6vE7NGMd1sAL4MV75O6O6ERD7VUeQFLRHdUwOv1CBpPLMcoU8oHq8 vZsGDVjKVwcSiEp8AzRh2jH9ld6O4aX73ngrmQUYxuSQVnj1yiiPljl0wXT+j5ecKXzk PEuTKw1A7UFwIxFBW/VB6+UuyjYlhlS3kCf3Q42+pRRWJgXlF0fxyEB14ZsCKZFdg2Al C3JQ==
X-Gm-Message-State: ALoCoQk5zQBzCROJtjAE3N2ZJWpN4rGrO0qyvMNMHruXX55CYvu2QWo6F2vNQVr4poVZISCwBNRO
X-Received: by 10.50.13.100 with SMTP id g4mr19180107igc.9.1397510585395; Mon, 14 Apr 2014 14:23:05 -0700 (PDT)
Received: from [192.168.0.197] (c-24-7-193-205.hsd1.il.comcast.net. [24.7.193.205]) by mx.google.com with ESMTPSA id on9sm32357946igb.11.2014.04.14.14.23.03 for <multiple recipients> (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 14 Apr 2014 14:23:04 -0700 (PDT)
User-Agent: Microsoft-MacOutlook/14.3.9.131030
Date: Mon, 14 Apr 2014 16:23:00 -0500
From: Carol Davids <davids@iit.edu>
To: William Cerveny <bmwg@wjcerveny.com>, bmwg@ietf.org
Message-ID: <CF71BBBE.5ACFC%davids@iit.edu>
Thread-Topic: [bmwg] My comments on draft-ietf-bmwg-sip-bench-meth-09
References: <1396883172.2235.103678713.65DD6B82@webmail.messagingengine.com>
In-Reply-To: <1396883172.2235.103678713.65DD6B82@webmail.messagingengine.com>
Mime-version: 1.0
Content-type: text/plain; charset="US-ASCII"
Content-transfer-encoding: 7bit
Archived-At: http://mailarchive.ietf.org/arch/msg/bmwg/UyhoQoAjtr_cgiwjESbq_5wysKE
Subject: Re: [bmwg] My comments on draft-ietf-bmwg-sip-bench-meth-09
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 14 Apr 2014 21:23:11 -0000
Thanks. We will review as soon as possible. Regards, Carol Carol Davids Professor & Director RTC Lab Illinois Institute of Technology Office: 630-682-6024 Mobile: 630-292-9417 Email: davids@iit.edu Skype: caroldavids1 Web: rtc-lab.itm.iit.edu On 4/7/14, 10:06 AM, "William Cerveny" <bmwg@wjcerveny.com> wrote: >Below are my high level comments on draft-ietf-bmwg-sip-bench-meth-09 >denoted by /* comment */ ; I'm sending more detailed (mostly grammatical >comments) directly to the authors. I've identified the section and >general line number in which the comment is made. > >Of note, I had trouble following the variables in the pseudocode, but I >may not have been following close enough attention to the code. > >Bill Cerveny > ><begin comments on draft-ietf-bmwg-sip-bench-meth-09> > >desktop-10-36:scratch wcerveny$ cat grep-output-meth-2014-04-17.txt >1. Abstract: > >30- objective comparison of the capacity of SIP devices. Test setup >31- parameters and a methodology are necessary because SIP allows a >wide >32- range of configuration and operational conditions that can >influence >33:/* In my opinion, sentence beginning with "A standard terminology >..." >34: is an assumed and I'm not sure it should be in the abstract */ >35- performance benchmark measurements. A standard terminology and >36- methodology will ensure that benchmarks have consistent definition >37- and were obtained following the same procedures. >-- >2. Introduction: >-- >200- only. >201- >202- The device-under-test (DUT) is a SIP server, which may be any SIP >203:/* Capitalization in "Benchmarks can be ..." is inconsistant */ >204- conforming [RFC3261] device. Benchmarks can be obtained and >compared >205- for different types of devices such as SIP Proxy Server, Session >206- Border Controllers (SBC), SIP registrars and SIP proxy server >paired >-- >-- >208- >209- The test cases provide metrics for benchmarking the maximum 'SIP >210- Registration Rate' and maximum 'SIP Session Establishment Rate' >that >211:/* Is "extended period" defined? */ >212- the DUT can sustain over an extended period of time without >failures. >213- Some cases are included to cover Encrypted SIP. The test >topologies >214- that can be used are described in the Test Setup section. >Topologies >-- >-- >219- >220- SIP permits a wide range of configuration options that are >explained >221- in Section 4 and Section 2 of [I-D.sip-bench-term]. Benchmark >values >222:/* Is associated media defined */ >223- could possibly be impacted by Associated Media. The selected >values >224- for Session Duration and Media Streams per Session enable >benchmark >225- >-- >3. Benchmarking Topologies >-- >259- >260- There are two test topologies; one in which the DUT does not >process >261- the media (Figure 1) and the other in which it does process media >262:/* EA defined? */ >263- (Figure 2). In both cases, the tester or EA sends traffic into >the >264- DUT and absorbs traffic from the DUT. The diagrams in Figure 1 >and >265- Figure 2 represent the logical flow of information and do not >dictate >-- >4.3 Associated Media >-- >346-4.3. Associated Media >347- >348- Some tests require Associated Media to be present for each SIP >349:/* Is this redundant? */ >350- session. The test topologies to be used when benchmarking DUT >351- performance for Associated Media are shown in Figure 1 and Figure >2. >352- >-- >4.6 Session Duration >-- >370- >371- The value of the DUT's performance benchmarks may vary with the >372- duration of SIP sessions. Session Duration MUST be reported with >373:/* I'm not sure if this sentence ("A Session Duration ...") is >properly >374-formed (it might be), but I had difficulty following the logic of >the >375:sentence */ >376- benchmarking results. A Session Duration of zero seconds >indicates >377- transmission of a BYE immediately following successful SIP >378- establishment indicate by receipt of a 200 OK. An infinite >Session >-- >4.8 Benchmarking algorithm >-- >409- >410- During the Candidate Identification phase, the test runs until n >411- sessions have been attempted, at session attempt rates, r, which >vary >412:/* Upper case N and lower case n are different variables?? Same with >"R" */ >413- according to the algorithm below, where n is also a parameter of >test >414- and is a relatively large number, but an order of magnitude >smaller >415- than N. If no errors occur during the time it takes to attempt n >-- >-- >415- than N. If no errors occur during the time it takes to attempt n >416- sessions, we increment r according to the algorithm. If errors >are >417- encountered during the test, we decrement r according to the >418:/* sentence "The algorithm provides ..." needs clarification >419:Is the word "how" unnecessary? */ >420- algorithm. The algorithm provides a variable, G, that allows us >to >421- control how the accuracy, in sessions per second, that we require >of >422- the test. >-- >-- >422- the test. >423- >424- After this candidate rate has been discovered, the test enters >the >425:/* Is N consistent with N in pseudocode? */ >426- Steady State phase. In the Steady State phase, N session >Attempts >427- are made at the candidate rate. The goal is to find a rate at >which >428- the DUT can process calls "forever" with no errors and the test >-- >-- >432- the steady-state phase is entered again until a final (new) >steady- >433- state rate is achieved. >434- >435:/* Would this process be clearer if presented as a list? */ >436- The iterative process itself is defined as follows: A starting >rate >437- of r = 100 sessions per second is used and we place calls at that >438- rate until n = 5000 calls have been placed. If all n calls are >-- >-- >436- The iterative process itself is defined as follows: A starting >rate >437- of r = 100 sessions per second is used and we place calls at that >438- rate until n = 5000 calls have been placed. If all n calls are >439:/* sps defined? This said, it's easy to figure out */ >440- successful, the rate is increased to 150 sps and again we place >calls >441- at that rate until n = 5000 calls have been placed. The attempt >rate >442- is continuously ramped up until a failure is encountered before n >= >-- >-- >449- between the rate at which failures occurred and the last >successful >450- rate. Continuing in this way, an attempt rate without errors is >451- found. The tester can specify a margin of error using the >parameter G, >452:/* units? */ >453- measured in units of sessions per second. >454- >455- The pseudo-code corresponding to the description above follows. >-- >-- >478- G := 5 ; granularity of results - the margin of error in >479- ; sps >480- C := 0.05 ; calibration amount: How much to back down if we >481:/* using "s" before definition, in my opinion; consider "found >candidate rate >482:s but cannot send at s" (Still not right, though ... */ >483- ; have found candidate s but cannot send at rate >s >484- ; for time T without failures >485- >-- >-- >487- ; ---- Initialization of flags, candidate values and upper >bounds >488- >489- f := false ; indicates a success after the upper limit >490:/* Capital F never used in pseudocode */ >491- F := false ; indicates that test is done >492- c := 0 ; indicates that we have found an upper limit >493- >-- >-- >499- ; characteristics until n >500- ; requests have been sent >501- if (all requests succeeded) { >502:/* undefined variable r'? does this matter? */ >503- r' := r ; save candidate value of metric >504- if ( c == 0 ) { >-- >6.3 Session Establishment Rate with Media not on DUT >-- >649- be recorded using any pertinent parameters as shown in the >650- reporting format of Section 5.1. >651- >652:/* Long sentence in general, but minimally last part of sentence >doesn't >653:conclude */ >654- Expected Results: Session Establishment Rate results obtained >with >655- Associated Media with any number of media streams per SIP >session >656- are expected to be identical to the Session Establishment Rate > ><end comments on draft-ietf-bmwg-sip-bench-meth-09> > >_______________________________________________ >bmwg mailing list >bmwg@ietf.org >https://www.ietf.org/mailman/listinfo/bmwg
- [bmwg] My comments on draft-ietf-bmwg-sip-bench-m… William Cerveny
- Re: [bmwg] My comments on draft-ietf-bmwg-sip-ben… Carol Davids