RE: [bmwg] Draft Agenda for bmwg at IETF-67

["Poretsky, Scott" <[e-mail]>]

Hi BMWG-ers,

Below is the Work Item Proposal submitted this past May for SIP Performance Benchmarking.  This proposed work item was already discussed at IETF 66 in the SIPPING and BMWG working groups.  The posted Terminology and Methodology (links below) can be reviewed for additional detail.  At this IETF 67 meeting we will review incorporated changes, revised scope, and methodologies.

www.ietf.org/internet-drafts/draft-poretsky-sip-bench-term-01.txt (-02 coming by Monday)

meth-00 was submitted last week but is still awaiting posting.

Scott
#############################

BMWG Work Item Proposal

TITLE: Session Initiation Protocol (SIP) Performance Benchmarking

AUTHOR TEAM:
Scott Poretsky, Reef Point Systems
Dr. Carol Davids, Illinois Institute of Technology
Dr. Vijay Gurbani, Lucent Technologies

GOALS
The purpose of this work item is to provide a single terminology and methodology from which SIP equipment vendors and VoIP service providers can measure performance benchmarking metrics for comparison and selection.  It is intended to develop terms, benchmarks, and methodologies that can be applied to any type of IP device including SIP Servers, Session Border Controllers (SBCs), and Security Gateways (SEGs).

MOTIVATION
Service Providers are now planning VoIP and Multimedia network deployments using the IETF developed Session Initiation Protocol (SIP).  Through SIP, service providers will be able to have rich service offerings and many even plan to turn off their Public Switch Telephone Network within five years.   VoIP has led to development of new networking devices including SIP Servers, Session Border Controllers, and Security Gateways.  The mix of voice and IP functions in this variety of devices has produced inconsistencies in vendor reported performance metrics and has caused confusion in the service provider community. 

SCOPE:
This work item will provide terms, benchmarks, and methodologies for performance benchmarking the SIP control and media planes.  The methodologies can be used for benchmarking SIP performance of SIP servers, SBCs, and SEGs.  The media used for the benchmarking will be the IETF defined Real-Time Protocol (RTP).  Test cases will allow testing of VoIP or Multimedia over IP by varying the number of media streams per SIP call from 1 to higher.  There will be at least one test case to measure the impact of a SIP DOS Attack on the performance metrics.  All SIP traffic will be in the clear; no encryption using TLS or IPsec will be used.  Similar benchmarks and methodologies using encryption is a potential work item for the BMWG that could be considered in the future.

There will be separate test cases in the methodology document for obtaining measurements for each of the Benchmarking Metrics defined in the Terminology.  The benchmarking metrics to be measured for the SIP Control Plane and SIP Media Plane are as follow:


Benchmark Metrics - SIP Control Plane 
Standing Calls, maximum (calls)
Calls Per Second, maximum (CPS)
Call Attempts Per Second, maximum (CAPS)
Busy Hour Call Attempts, maximum (BHCA)
Busy Hour Call Connects, maximum (BHCC)
Call Completion Rate (%)
Call Setup Delay, average (msec)
Call Teardown Delay, average (msec)

Benchmark Metrics- SIP Media Plane
RTP Media Throughput, per Media Stream (pps)
RTP Media Throughput, Aggregate (pps)
RTP Packet Loss, average (pps)
RTP Packet Delay, average (msec)
RTP Packet Jitter, average (msec)

In order to obtain measurements for the benchmarking metrics it will be necessary to configure the test setup with certain test parameters.  Some of these test parameters may also be the benchmarking metric being measured for another test case.  The test parameters to be configured for the SIP Control Plane and SIP Media Plane are as follow:

Test Parameters- SIP Control Plane 
Call Duration (msec)
Call Per Second (CPS)
Call Attempts Per Second (CAPS)

Test Parameters - SIP Media Plane
RTP Media Streams per Call (streams per call)
RTP Packet Size (bytes)
RTP Media Offered Load, per Media Stream (pps)
RTP Media Offered Load, Aggregate (pps)

The basic test topology to be used for this benchmarking is as follows:

Emulated Agents<--> SEG (optional) <à SIP Proxy Server (or SBC) <--> SIP Server (optional) <---> Emulated Agents

PROPOSED MILESTONES
06/07 First Draft Terminology and Methodology
02/07 First WG Last Call
05/07 SIP WG and SIPPING WG Review
06/07 Final BMWG Last Call for Terminology and Methodology
08/07 Submittal for IESG review


-----Original Message-----
From: Al Morton [mailto:[e-mail]] 
Sent: Friday, October 20, 2006 9:34 AM
To: [e-mail]
Subject: [bmwg] Draft Agenda for bmwg at IETF-67


Benchmarking Methodology WG (bmwg)

Tuesday, Afternoon Session I 1300-1500
Room Name: Nautilus 1

CHAIR(s): Al Morton <[e-mail]>

AGENDA:

0.  Agenda bashing (if we need to shuffle a few items)
     See 
https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=67
     for Agenda updates and Slides

1.  Working Group Status (Chair)

Topics/Drafts not covered by presentations below:

      Hash and Stuffing Draft (WGLC completed)
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-hash-stuffing-06.txt

      Terms and Methods for Benchmarking IPsec Devices
               (WG review and feedback is needed! Please read and comment...)
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-ipsec-term-08.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-ipsec-meth-01.txt
(Recently Expired, see http://tools.ietf.org/wg/bmwg/ for these versions)

      Methodology for Benchmarking Network-layer Traffic Control Mechanisms
http://tools.ietf.org/wg/bmwg/draft-ietf-bmwg-dsmmeth/draft-ietf-bmwg-dsmmeth-02.txt

     IGP Data plane convergence benchmark I-Ds - Publication Requested

     Standard Introduction/Security Paragraph for BMWG memos

Check the BMWG mail archive for comments:
      http://www.ietf.org/mail-archive/working-groups/bmwg/current/


2.   Techniques for Benchmarking Core Router Accelerated Stress Testing.

Discussion GOALS: Review Last Call responses (numerous from many
reviewers), updates to new revisions, and open issues
Draft file name, or preferably the complete URL:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-10.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-06.txt
draft-ietf-bmwg-acc-bench-meth-opsec-00
Presenter: Scott Poretsky


3.   IPv6 Benchmarking Methodology

Discussion GOALS:
a) Updates based on BMWG and v6ops feedback
- Includes recommendations for benchmarking with traffic that has the
   Hop-by-Hop Extension Header (as suggested by v6ops and BMWG)
- Updated frame size recommendations for Ethernet to include Jumbo frames
   (Appendix updated to include line rates values for all frame sizes)
- Proposal for IPv6 benchmarking prefix assignment by IANA
- The Security section contains the new standard text discussed within BMWG
   (we can take this opportunity to discuss the Security write-up)
b) Call for making the document a WG work item and discuss next steps
Presenter: Gunter van de Velde, Diego Dugatkin, or Ahmed Hamza
(one of these three gentlemen will be presenting)

Draft:
http://tools.ietf.org/wg/bmwg/draft-popoviciu-bmwg-ipv6benchmarking-02.txt


4.   Sub-IP Protection Mechanisms

Discussion GOALS: Introduce this new official work item and review
changes in the -00 terminology and methodology documents different from
the individual submittals.
Presenter: Scott Poretsky

Drafts:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-protection-term-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-protection-meth-00.txt


5.   Milestone Status (Chair)


*******   New Work Proposals  *********


6.    SIP Performance Benchmarking

Discussion GOALS: Review changes to Terminology, review new Methodology.
Ask working group if ready to be an official work item.
Presenter: Scott Poretsky

Related Drafts:
draft-poretsky-bmwg-sip-bench-term-01.txt
draft-poretsky-bmwg-sip-bench-meth-00.txt


7.   Multicast VPN Scalability Benchmarking

Discussion GOALS:
- Introduce the problem of MVPN scalability benchmarking and need for
this draft
- Introduce MVPN Metric proposal
- Summarize proposed methodology
- Request BMWG to take it on as WG work item


Statement of proposed work:
    Multicast VPN (MVPN) is a service deployed by VPN service providers
    to enable their customers to use IP multicast applications over VPNs.

    With the increased popularity the scalability of deploying such a
    service is becoming of a great interest. This document defines
    standard metric and test methodology for characterizing and comparing
    control plane MVPN scalability of Provider Edge (PE) devices that
    implement MVPN service.

Presenter: Silvija Dry

Related Draft:
http://www.ietf.org/internet-drafts/draft-sdry-bmwg-mvpnscale-00.txt


8.   Extending the current methodologies to cover wireless LAN switches and
wireless LAN meshes

TOPIC: WLAN Switch and Mesh Benchmarking
GOALS: Provide background
        Propose specific work items
        Solicit help towards generating a candidate draft
DRAFTS: None
PRESENTER: Jerry Perser, Tom Alexander

Statement of Proposed Work
------------------------------
Enterprise wireless LANs are now comprised of highly IP-centric devices.
Considerable work is being done in various IETF WGs in this area. For example,
CAPWAP is defining WLAN switch protocols, and MANET is defining IP over
wireless mesh and ad-hoc networks. Modern WLAN switches are Layer 3/4 aware
and include many traditional IETF defined functions such as ARP, DHCP and
firewalling in combination with wireless functions such as mobility. 
The lack of
industry-standard metrics for benchmarking makes it very difficult 
for vendors to
compare and improve the performance of their devices.


We propose that BMWG should take up work in support of these technologies.
Metrics are required for roaming and scalability performance (related to
CAPWAP) and multi-hop performance and recovery times (related to MANET),
as well as general WLAN switch data plane performance. The proposed
work item is as follows:
         - extend existing LAN switch benchmarking terminology and
                 methodology (RFC 1242, RFC 2285 etc.) to wireless
                 LAN switching devices
         - create new wireless-specific terminology and methodology
                 for mobility, scalability, and mesh networks


The IEEE 802.11T Task Group has determined that such work does not fall
within their scope, and so there should be no issue with taking it up in BMWG.


Scope
-------
The scope of the proposed work will cover benchmarking terminology and
methodology for measuring the performance of wireless LAN switches and
wireless LAN meshes, with particular focus on extending RFC 1242, RFC
2544, RFC 2285 and RFC 2889 to such systems. Benchmarking and
performance of 802.11 PHY or link layer functions is a non-objective.


Status of existing drafts
-------------------------
No drafts currently exist in support of this work.


For background on related BMWG wireless discussions, see:
http://www3.ietf.org/proceedings/04nov/bmwg.html#cmr



Please help contribute to a successful meeting by reading the above
I-D(s) and references *before* we meet.

To offer comments on BMWG work in progress or the agenda itself,
please send email to:

               [e-mail]

Alternatively, to offer potential agenda items, please email:

               [e-mail]


_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg