Re: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)
"MORTON, ALFRED C (AL)" <acmorton@att.com> Thu, 12 June 2014 18:21 UTC
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 054F01B27F5 for <bmwg@ietfa.amsl.com>; Thu, 12 Jun 2014 11:21:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.852
X-Spam-Level:
X-Spam-Status: No, score=-4.852 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651, SPF_HELO_PASS=-0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VB8KAzg2n4Sl for <bmwg@ietfa.amsl.com>; Thu, 12 Jun 2014 11:21:06 -0700 (PDT)
Received: from mail-red.research.att.com (mail-red.research.att.com [204.178.8.23]) by ietfa.amsl.com (Postfix) with ESMTP id 2EB661B27BB for <bmwg@ietf.org>; Thu, 12 Jun 2014 11:21:06 -0700 (PDT)
Received: from mail-green.research.att.com (H-135-207-255-15.research.att.com [135.207.255.15]) by mail-red.research.att.com (Postfix) with ESMTP id 687D7554AEC for <bmwg@ietf.org>; Thu, 12 Jun 2014 14:24:06 -0400 (EDT)
Received: from njfpsrvexg8.research.att.com (unknown [135.207.255.243]) by mail-green.research.att.com (Postfix) with ESMTP id 11FABE2DBD for <bmwg@ietf.org>; Thu, 12 Jun 2014 14:20:21 -0400 (EDT)
Received: from NJFPSRVEXG8.research.att.com ([fe80::cdea:b3f6:3efa:1841]) by njfpsrvexg8.research.att.com ([fe80::cdea:b3f6:3efa:1841%13]) with mapi; Thu, 12 Jun 2014 14:21:05 -0400
From: "MORTON, ALFRED C (AL)" <acmorton@att.com>
To: bmwg WG <bmwg@ietf.org>
Date: Thu, 12 Jun 2014 14:21:05 -0400
Thread-Topic: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)
Thread-Index: Ac+GaE8YL3WjkqoqThib3obvweDfgAAAlgNQ
Message-ID: <2845723087023D4CB5114223779FA9C801896A7842@njfpsrvexg8.research.att.com>
References: <20140612180054.16608.72680.idtracker@ietfa.amsl.com>
In-Reply-To: <20140612180054.16608.72680.idtracker@ietfa.amsl.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: http://mailarchive.ietf.org/arch/msg/bmwg/IcHAL28isSma3bCyrGFGID2JiZg
Subject: Re: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 12 Jun 2014 18:21:08 -0000
We made it this far! Everything we asked to do is in there. Let's get to work and meet our milestones. Al bmwg co-chair > -----Original Message----- > From: bmwg [mailto:bmwg-bounces@ietf.org] On Behalf Of The IESG > Sent: Thursday, June 12, 2014 2:01 PM > To: IETF-Announce > Cc: bmwg WG > Subject: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg) > > The Benchmarking Methodology (bmwg) working group in the Operations and > Management Area of the IETF has been rechartered. For additional > information please contact the Area Directors or the WG Chairs. > > Benchmarking Methodology (bmwg) > ------------------------------------------------ > Current Status: Active WG > > Chairs: > Sarah Banks <sbanks@akamai.com> > Al Morton <acmorton@att.com> > > Assigned Area Director: > Joel Jaeggli <joelja@bogus.com> > > Mailing list > Address: bmwg@ietf.org > To Subscribe: bmwg-request@ietf.org > Archive: http://www.ietf.org/mail-archive/web/bmwg/ > > Charter: > > The Benchmarking Methodology Working Group (BMWG) will continue to > produce a series of recommendations concerning the key performance > characteristics of internetworking technologies, or benchmarks for > network devices, systems, and services. Taking a view of networking > divided into planes, the scope of work includes benchmarks for the > management, control, and forwarding planes. > > Each recommendation will describe the class of equipment, system, or > service being addressed; discuss the performance characteristics that > are pertinent to that class; clearly identify a set of metrics that aid > in the description of those characteristics; specify the methodologies > required to collect said metrics; and lastly, present the requirements > for the common, unambiguous reporting of benchmarking results. > > The set of relevant benchmarks will be developed with input from the > community of users (e.g., network operators and testing organizations) > and from those affected by the benchmarks when they are published > (networking and test equipment manufacturers). When possible, the > benchmarks and other terminologies will be developed jointly with > organizations that are willing to share their expertise. Joint review > requirements for a specific work area will be included in the detailed > description of the task, as listed below. > > To better distinguish the BMWG from other measurement initiatives in the > IETF, the scope of the BMWG is limited to the characterization of > implementations of various internetworking technologies > using controlled stimuli in a laboratory environment. Said differently, > the BMWG does not attempt to produce benchmarks for live, operational > networks. Moreover, the benchmarks produced by this WG shall strive to > be vendor independent or otherwise have universal applicability to a > given technology class. > > Because the demands of a particular technology may vary from deployment > to deployment, a specific non-goal of the Working Group is to define > acceptance criteria or performance requirements. > > An ongoing task is to provide a forum for development and > advancement of measurements which provide insight on the > capabilities and operation of implementations of inter-networking > technology. > > Ideally, BMWG should communicate with the operations community > through organizations such as NANOG, RIPE, and APRICOT. > > The BMWG is explicitly tasked to develop benchmarks and methodologies > for the following technologies: > > BGP Control-plane Convergence Methodology (Terminology is complete): > With relevant performance characteristics identified, BMWG will prepare > a Benchmarking Methodology Document with review from the Routing Area > (e.g., the IDR working group and/or the RTG-DIR). The Benchmarking > Methodology will be Last-Called in all the groups that previously > provided input, including another round of network operator input during > the last call. > > SIP Networking Devices: Develop new terminology and methods to > characterize the key performance aspects of network devices using > SIP, including the signaling plane scale and service rates while > considering load conditions on both the signaling and media planes. This > work will be harmonized with related SIP performance metric definitions > prepared by the PMOL working group. > > Traffic Management: Develop the methods to characterize the capacity > of traffic management features in network devices, such as > classification, > policing, shaping, and active queue management. Existing terminology > will be used where appropriate. Configured operation will be verified > as a part of the methodology. The goal is a methodology to assess the > maximum forwarding performance that a network device can sustain without > dropping or impairing packets, or compromising the accuracy of multiple > instances of traffic management functions. This is the benchmark for > comparison between devices. Another goal is to devise methods that > utilize flows with congestion-aware transport as part of the traffic > load and still produce repeatable results in the isolated test > environment. > > IPv6 Neighbor Discovery: Large address space in IPv6 subnets presents > several networking challenges, as described in RFC 6583. Indexes to > describe the performance of network devices, such as the number of > reachable devices on a sub-network, are useful benchmarks to the > operations community. The BMWG will develop the necessary > terminology and methodologies to measure such benchmarks. > > In-Service Software Upgrade: Develop new methods and benchmarks to > characterize the upgrade of network devices while in-service, > considering both data and control plane operations and impacts. > These devices are generally expected to maintain control plane session > integrity, including routing connections. Quantification of upgrade > impact will include packet loss measurement, and other forms of recovery > behavior will be noted accordingly. The work will produce a definition > of ISSU, which will help refine the scope.??Liaisons will be established > as needed. > > Data Center Benchmarking: This work will define additional terms, > benchmarks, and methods applicable to data center performance > evaluations. > This includes data center specific congestion scenarios, switch buffer > analysis, microburst, head of line blocking, while also using a wide mix > of traffic conditions. Some aspects from BMWG's past work are not > meaningful when testing switches that implement new IEEE specifications > in the area of data center bridging. For example, throughput as defined > in RFC 1242 cannot be measured when testing devices that implement three > new IEEE specifications: priority-based flow control (802.1Qbb); > priority groups (802.1Qaz); and congestion notification (802.1Qau). > This work will update RFC1242, RFC2544, RFC2889 (and other key RFCs), > and exchange Liaisons with relevant SDOs, especially at WG Last Call. > > VNF and Related Infrastructure Benchmarking: Benchmarking Methodologies > have reliably characterized many physical devices. This work item extends > > and enhances the methods to virtual network functions (VNF) and their > unique supporting infrastructure. A first deliverable from this activity > will be a document that considers the new benchmarking space to ensure > that common issues are recognized from the start, using background > materials from industry and SDOs (e.g., IETF, ETSI NFV). > Benchmarks for platform capacity and performance characteristics of > virtual routers, switches, and related components will follow, including > comparisons between physical and virtual network functions. In many > cases, > the traditional benchmarks should be applicable to VNFs, but the lab > set-ups, configurations, and measurement methods will likely need to > be revised or enhanced. > > Milestones: > Jun 2014 - Basic BGP Convergence Benchmarking Methodology to IESG > Review > Jul 2014 - Terminology for SIP Device Benchmarking to IESG Review > Jul 2014 - Methodology for SIP Device Benchmarking to IESG Review > Aug 2014 - Draft on Traffic Management Benchmarking to IESG Review > Dec 2014 - Draft on IPv6 Neighbor Discovery to IESG Review > Mar 2015 - Draft on In-Service Software Upgrade Benchmarking to IESG > Review > Aug 2015 - Draft on VNF Benchmarking Considerations to IESG Review > Dec 2015 - Drafts on Data Center Benchmarking to IESG Review >
- Re: [bmwg] WG Action: Rechartered Benchmarking Me… MORTON, ALFRED C (AL)
- [bmwg] WG Action: Rechartered Benchmarking Method… The IESG
- Re: [bmwg] WG Action: Rechartered Benchmarking Me… Barry Constantine
- Re: [bmwg] WG Action: Rechartered Benchmarking Me… Banks, Sarah