Re: [bmwg] Re-Charter Text for review

[joel jaeggli <[e-mail]>]

Thank you,

I think we can work with that as a proposal for the IESG.

While it's not directly the basis for evaluation we should sketch out
the milestones that in the near term we are undertaking against the
charter items.

joel

On 5/3/14, 3:20 PM, MORTON, ALFRED C (AL) wrote:
> Description of Working Group:
> 
> 
> The Benchmarking Methodology Working Group (BMWG) will continue to
> produce a series of recommendations concerning the key performance
> characteristics of internetworking technologies, or benchmarks for
> network devices, systems, and services. Taking a view of networking
> divided into planes, the scope of work includes benchmarks for the
> management, control, and forwarding planes.
> 
> Each recommendation will describe the class of equipment, system, or
> service being addressed; discuss the performance characteristics that
> are pertinent to that class; clearly identify a set of metrics that aid
> in the description of those characteristics; specify the methodologies
> required to collect said metrics; and lastly, present the requirements
> for the common, unambiguous reporting of benchmarking results.
> 
> The set of relevant benchmarks will be developed with input from the
> community of users (e.g., network operators and testing organizations)
> and from those affected by the benchmarks when they are published
> (networking and test equipment manufacturers). When possible, the
> benchmarks and other terminologies will be developed jointly with
> organizations that are willing to share their expertise. Joint review
> requirements for a specific work area will be included in the detailed
> description of the task, as listed below.
> 
> To better distinguish the BMWG from other measurement initiatives in the
> IETF, the scope of the BMWG is limited to the characterization of
> implementations of various internetworking technologies
> using controlled stimuli in a laboratory environment. Said differently,
> the BMWG does not attempt to produce benchmarks for live, operational
> networks. Moreover, the benchmarks produced by this WG shall strive to
> be vendor independent or otherwise have universal applicability to a
> given technology class.
> 
> Because the demands of a particular technology may vary from deployment
> to deployment, a specific non-goal of the Working Group is to define
> acceptance criteria or performance requirements.
> 
> An ongoing task is to provide a forum for development and
> advancement of measurements which provide insight on the
> capabilities and operation of implementations of inter-networking technology.
> 
> The BMWG will communicate with the operations community through
> organizations such as NANOG, RIPE, and APRICOT.
> 
> In addition to its current work items, the BMWG is explicitly tasked to
> develop benchmarks and methodologies for the following technologies:
> 
> Traffic Management: Develop the methods to characterize the capacity 
> of traffic management features in network devices, such as classification, 
> policing, shaping, and active queue management. Existing terminology 
> will be used where appropriate. Configured operation will be verified 
> as a part of the methodology. The goal is a methodology to assess the 
> maximum forwarding performance that a network device can sustain without 
> dropping or impairing packets, or compromising the accuracy of multiple 
> instances of traffic management functions. This is the benchmark for 
> comparison between devices. Another goal is to devise methods that 
> utilize flows with congestion-aware transport as part of the traffic 
> load and still produce repeatable results in the isolated test environment.
> 
> IPv6 Neighbor Discovery: Large address space in IPv6 subnets presents 
> several networking challenges, as described in RFC 6583. Indexes to 
> describe the performance of network devices, such as the number of 
> reachable devices on a sub-network, are useful benchmarks to the 
> operations community. The BMWG will develop the necessary 
> terminology and methodologies to measure such benchmarks.
> 
> In-Service Software Upgrade: Develop new methods and benchmarks to 
> characterize the upgrade of network devices while in-service, 
> considering both data and control plane operations and impacts. 
> These devices are generally expected to maintain control plane session 
> integrity, including routing connections. Quantification of upgrade 
> impact will include packet loss measurement, and other forms of recovery 
> behavior will be noted accordingly. The work will produce a definition 
> of ISSU, which will help refine the scope. Liaisons will be established 
> as needed.
> 
> Data Center Benchmarking: This work will define additional terms, 
> benchmarks, and methods applicable to data center performance evaluations. 
> This includes data center specific congestion scenarios, switch buffer 
> analysis, microburst, head of line blocking, while also using a wide mix 
> of traffic conditions. Some aspects from BMWG's past work are not 
> meaningful when testing switches that implement new IEEE specifications 
> in the area of data center bridging. For example, throughput as defined 
> in RFC 1242 cannot be measured when testing devices that implement three 
> new IEEE specifications: priority-based flow control (802.1Qbb); 
> priority groups (802.1Qaz); and congestion notification (802.1Qau).
> This work will update RFC1242, RFC2544, RFC2889 (and other key RFCs), 
> and exchange Liaisons with relevant SDOs, especially at WG Last Call.
> 
> VNF and Related Infrastructure Benchmarking: Benchmarking Methodologies 
> have reliably characterized many physical devices. This work item extends 
> and enhances the methods to virtual network functions (VNF) and their 
> unique supporting infrastructure. First, the new task space will be 
> considered to ensure that common issues are recognized from the start. 
> Benchmarks for platform capacity and performance characteristics of 
> virtual routers, switches, and related components will follow, including 
> comparisons between physical and virtual network functions.
> 
>