Re: [bmwg] Revised Draft Text for Re-Chartering

["MORTON, ALFRED C (AL)" <[e-mail]>]

BMWG,

We have slightly revised the Proposed Milestones as part of 
Re-Charter Review in our IETF-101 session.

Revised proposal is below.

Final Comment period is one week, by March 21st, 2018, to the
bmwg-list please.

regards,
Al
bmwg co-chair

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
BMWG - Benchmarking Methodology Working Group  (as of March 20, 2018)

Description of Working Group:

  The Benchmarking Methodology Working Group (BMWG) will continue to
  produce a series of recommendations concerning the key performance
  characteristics of internetworking technologies, or benchmarks for
  network devices, systems, and services. Taking a view of networking
  divided into planes, the scope of work includes benchmarks for the
  management, control, and forwarding planes. 

  The scope of BMWG has been extended to develop methods for
  virtual network functions (VNF) and their unique supporting 
  infrastructure (such as SDN Controllers and vSwitches). 
  Benchmarks for platform capacity and performance characteristics 
  of virtual routers, firewalls (and other security functions), 
  signaling control gateways, and other forms of gateways are included.  
  The benchmarks will foster comparisons between physical and virtual 
  network functions, and also cover unique features of  
  Network Function Virtualization systems. Also, with the emergence
  of virtualized test systems, specifications for test system
  calibration are also in-scope.

  Each recommendation will describe the class of network function, system, 
  or service being addressed; discuss the performance characteristics that
  are pertinent to that class; clearly identify a set of metrics that aid
  in the description of those characteristics; specify the methodologies
  required to collect said metrics; and lastly, present the requirements
  for the common, unambiguous reporting of benchmarking results.

  The set of relevant benchmarks will be developed with input from the
  community of users (e.g., network operators and testing organizations)
  and from those affected by the benchmarks when they are published
  (networking and test equipment manufacturers). When possible, the
  benchmarks and other terminologies will be developed jointly with
  organizations that are willing to share their expertise. Joint review
  requirements for a specific work area will be included in the
  description of the task.

  To better distinguish the BMWG from other measurement initiatives in the
  IETF, the scope of the BMWG is limited to the characterization of
  implementations of various internetworking technologies
  using controlled stimuli in a laboratory environment. Said differently,
  the BMWG does not attempt to produce benchmarks for live, operational
  networks. Moreover, the benchmarks produced by this WG shall strive to
  be vendor independent or otherwise have universal applicability to a
  given technology class.

  Because the demands of a particular technology may vary from deployment
  to deployment, a specific non-goal of the Working Group is to define
  acceptance criteria or performance requirements.

  An ongoing task is to provide a forum for development and
  advancement of measurements which provide insight on the
  capabilities and operation of implementations of inter-networking 
  technology. Ideally, BMWG should communicate with the operations 
  community through organizations such as NANOG, RIPE, and APRICOT.

 
Goals and Milestones: (preliminary list, we have an active draft for all topics)

  Aug 2018 - Methodology for Next-Gen Firewall Benchmarking to IESG Review
  Dec 2018 - Update to RFC2544 Back-to-back Frame Benchmarking to IESG Review
  Dec 2019 - Methodology for EVPN Benchmarking to IESG Review
  Dec 2019 - Considerations for Benchmarking Network Virtualization Platforms 
             to IESG Review
  Dec 2019 - Draft on Network Service Layer Abstract Models applied to 
             Benchmarking to IESG Review
  Dec 2019 - Draft on General VNF Benchmarking Automation to IESG Review


> -----Original Message-----
> From: bmwg [mailto:[e-mail]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Thursday, March 1, 2018 10:13 AM
> To: [e-mail]
> Subject: [bmwg] Revised Draft Text for Re-Chartering
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> 
> BMWG,
> 
> We have slightly revised text to review during our
> Virtual Interim Meeting today (the first draft text
> was sent to bmwg-list in October, 2017).
> I'll summarize the changes during the meeting.
> 
> regards,
> Al
> bmwg-co-chair
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 
> BMWG - Benchmarking Methodology Working Group
> 
> Description of Working Group:
> 
>   The Benchmarking Methodology Working Group (BMWG) will continue to
>   produce a series of recommendations concerning the key performance
>   characteristics of internetworking technologies, or benchmarks for
>   network devices, systems, and services. Taking a view of networking
>   divided into planes, the scope of work includes benchmarks for the
>   management, control, and forwarding planes.
> 
>   The scope of BMWG has been extended to develop methods for
>   virtual network functions (VNF) and their unique supporting
>   infrastructure (such as SDN Controllers and vSwitches).
>   Benchmarks for platform capacity and performance characteristics
>   of virtual routers, firewalls (and other security functions),
>   signaling control gateways, and other forms of gateways are included.
>   The benchmarks will foster comparisons between physical and virtual
>   network functions, and also cover unique features of
>   Network Function Virtualization systems. Also, with the emergence
>   of virtualized test systems, specifications for test system
>   calibration are also in-scope.
> 
>   Each recommendation will describe the class of network function, system,
>   or service being addressed; discuss the performance characteristics that
>   are pertinent to that class; clearly identify a set of metrics that aid
>   in the description of those characteristics; specify the methodologies
>   required to collect said metrics; and lastly, present the requirements
>   for the common, unambiguous reporting of benchmarking results.
> 
>   The set of relevant benchmarks will be developed with input from the
>   community of users (e.g., network operators and testing organizations)
>   and from those affected by the benchmarks when they are published
>   (networking and test equipment manufacturers). When possible, the
>   benchmarks and other terminologies will be developed jointly with
>   organizations that are willing to share their expertise. Joint review
>   requirements for a specific work area will be included in the
>   description of the task.
> 
>   To better distinguish the BMWG from other measurement initiatives in the
>   IETF, the scope of the BMWG is limited to the characterization of
>   implementations of various internetworking technologies
>   using controlled stimuli in a laboratory environment. Said differently,
>   the BMWG does not attempt to produce benchmarks for live, operational
>   networks. Moreover, the benchmarks produced by this WG shall strive to
>   be vendor independent or otherwise have universal applicability to a
>   given technology class.
> 
>   Because the demands of a particular technology may vary from deployment
>   to deployment, a specific non-goal of the Working Group is to define
>   acceptance criteria or performance requirements.
> 
>   An ongoing task is to provide a forum for development and
>   advancement of measurements which provide insight on the
>   capabilities and operation of implementations of inter-networking
>   technology. Ideally, BMWG should communicate with the operations
>   community through organizations such as NANOG, RIPE, and APRICOT.
> 
> 
> Goals and Milestones: (preliminary list, we have an active draft for all
> topics)
> 
>   Dec 2018 - Methodology for Next-Gen Firewall Benchmarking to IESG Review
>   Dec 2018 - Update to RFC2544 Back-to-back Frame Benchmarking to IESG
> Review
>   Dec 2019 - Methodology for EVPN Benchmarking to IESG Review
>   Dec 2019 - Considerations for Benchmarking Network Virtualization
> Platforms
>              to IESG Review
>   Dec 2019 - Draft on Network Service Layer Abstract Models applied to
>              Benchmarking to IESG Review
>   Dec 2019 - Draft on Automated VNF Benchmarking to IESG Review
> 
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=lB6OG6NJlVPcNq_hFaAplcHHWP5AQ
> v2kMcsxYDTMFDo&s=xE6icgQqHQn-lu-aS00Fu45JJF6kvx92rmWZFwP1XFM&e=