[bmwg] BMWG minutes for IETF-58 proceedings

[Al Morton <[e-mail]>]

BMWG,

Here are the minutes as submitted to the proceedings,
our thanks to the note-takers and those who made comments.
The slides have been temporarily posted here:
http://www.ietf.org/proceedings/03nov/index.html

Kevin/Al


Benchmarking Methodology WG (bmwg)


Wednesday, November 12, 2003, 1300-1500
=======================================


CHAIRS Kevin Dubray <[e-mail]>
            Al Morton <[e-mail]>


Reported by Al Morton and Kevin Dubray, using information
generously compiled by Scott Poretsky and Tony DeLaRosa
as official note-takers.


About 25 people attended the BMWG session.

The session's agenda was approved as presented.

1.   Working Group Status (Morton)

The status of various BMWG I-Ds and proposals were reported thusly:

AD/IESG Review
      <draft-ietf-bmwg-conterm-05.txt>, revised, under review.
      <draft-ietf-bmwg-mcastm-13.txt>, OPS Directorate Comments
      <draft-ietf-bmwg-ospfconv-term-06.txt>,  OPS Dir Comments
      <draft-ietf-bmwg-ospfconv-intraarea-06.txt>, same
      <draft-ietf-bmwg-ospfconv-applicability-03.txt>, same
I-D Last Call
      <draft-ietf-bmwg-fib-meth-01.txt>,  Call ended 3/14.
      <draft-ietf-bmwg-dsmterm-08.txt>, Call ended 11/7, w/comment.
I-Ds
      <draft-ietf-bmwg-ipsec-term-02.txt>, draft 10/2003, comments@57
      <draft-ietf-bmwg-benchres-term-04.txt>, back in WG, comments@57
      <draft-ietf-bmwg-acc-bench-term-01.txt> Revised on comments
      <draft-ietf-bmwg-acc-bench-framework-00.txt> New
      <draft-ietf-bmwg-igp-dataplane-conv-term-01.txt> Revised
      <draft-ietf-bmwg-igp-dataplane-conv-meth-01.txt> Revised
      <draft-ietf-bmwg-igp-dataplane-conv-app-01.txt> Revised
Expired BMWG I-Ds
      <draft-ietf-bmwg-bgpbas-01.txt>, Pending term. progress
      <draft-ietf-bmwg-benchres-method-00.txt> Pending term progress
New Work proposals.
      <draft-kimura-protection-term-02.txt>
      <draft-poretsky-mpls-protection-meth-01.txt>

The old BMWG I-D on FIB benchmarking methodology I-D (expired) seems to
be dying a quiet death.  It was one of the original I-Ds to address
convergence.  The WG needs to decide whether this work is critical to the
body of convergence work or not.

-------------------------------------------------------------------------
2.   Benchmarking Network-layer Traffic Control Mechanisms:
        Terminology -- Communicate latest changes & results of
        WG Last Call ending Nov 7th. (J. Perser et al.)

        http://www.ietf.org/internet-drafts/draft-ietf-bmwg-dsmterm-08.txt
        http://www.ietf.org/mail-archive/working-groups/bmwg/current/

Jerry Perser presented slides (see proceedings) outlining the changes
reflected in the latest version of the draft as well as issues surrounding
the term "Channel Capacity".  Jerry briefly articulated the notion
of "forwarding capacity," as a possible successor to the channel capacity
term.

Jerry hopes to get the wording incorporated into the I-D quickly.  A
last call will be issued subsequent to the I-D's announcement.

-------------------------------------------------------------------------
3.  IPsec Device Benchmarking Terminology I-D. New developments in 02.
       Finalize recent list discussions on the packet mix possibilities;
       this topic has relevance to many other efforts. (A. Morton)

       http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ipsec-term-02.txt
       http://www.ietf.org/mail-archive/working-groups/bmwg/current/

Al thanked Michele Bustos, Tim Van Herck and Merike Kaeo
for preparing slide material.  The question at
the top of the docket for this draft was "IMIX", or the specifying of
an offered load composed of something other than a series of packets having
the same size. Three possible alternatives for "mixes" were presented
to the working group.

Commentary was offered that while this discussion was beneficial, it
doesn't appear exclusively bound to IPsec benchmarking.  It was suggested
that the issue be handled outside the scope of the IPsec benchmarking, in a
more general BMWG context.  Heads nodded.

There was concern expressed that any single mix of packet sizes would
not be typical for all classes of users. For example, the NLANR notion of
IMIX might not be typical for users of, say, VPNs.  It was countered that
the specification could allow a generalized packet mix
(hereafter referred to as a "mix") to be specified locally by
the testing body, with strict requirements on mix reporting.

It was pointed out that local specification of mix may short-circuit the
notion "neutrality" that is very desirable in benchmarks.  That is to say,
a "mix" while adequately documented, could still be biased based on the
mix's composition.

So, it was argued, viable solutions might be: a) give everyone a single
mix, or b) give no one a mix (as BMWG has done to date).

If, on the other hand, locally defined mixes were deemed acceptable, the group
believed that the mix option requiring strict reporting constraints must
be in place. It was pointed out that a mix composed of an increasing sweep
of packet sizes is a subset of the locally defined "mix" option.

It was again articulated this discussion should happen independent of the
IPsec terminology work.


-------------------------------------------------------------------------
4.  IGP Data plane convergence benchmark I-Ds.
      Changes from 00 to 01 to address issues from mailing list.
      Ready for Last Call?  Authors believe these are ready.
      (S.Poretsky)

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-01.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-01.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-01.txt

Scott presented his slides which outlined changes to the I-Ds and new
terms added.

One of the new terms, "Packet Sampling Interval", essentially sets
the time resolution for re-convergence measurements.
There was a comment that the draft should have recommendations
for the appropriate intervals to use with respect to the measured
convergence time interval.  Scott described the existing recommendations
using 0.1 second Packet Sampling Interval. The 01 terminology and
methodology drafts provide some detail, but there was interest
expressed in more explicit recommendations in this area.

Much time was spent on the slide which illustrated various terms
that describe the dataplane's response to failure/re-convergence
(e.g., Convergence Event Instant, Convergence Event Transition,
Convergence Recovery Transition, Convergence Recovery Instant, etc.)
on their relation with each other. (Slide 4 in this talk, see proceedings.)

A question was posed on how do you capture the graph of Slide 4
in the methodology.  One brand of test equipment produced
this plot, but not all test equipment will produce a graph like this,
and graphing the results is an exercise left to the reader.
Scott replied that all the required components are in the reporting
section of the Methodology I-D and defined in the Terminology ID
(where a tabular format is defined).

There was some discussion regarding the manner to capture the specific times
of rate stability (or instability).  Scott said the methodology maintains a
continuous survey for which there are 2 major constraints: 1) do you
have full convergence, 2) how long do you sustain that convergence.

Another question was asked regarding how would a reduction in
frame rate following convergence affect the convergence time.
Scott replied the convergence time approaches infinity in this case
(perhaps the DUT never achieves Full Convergence).

Regarding convergence recovery instant, how does one determine the
precise times that there are no longer packet losses?  It was thought
that the return to "full frame rate" would reflect that instant
(the definition of Rate-Derived Convergence time will be revised).
It was offered that it appeared that the indicators for convergence need
to be better defined, since one needs multiple measurements over time
to declare stability.

In summary, use of Forwarding Rate in the definitions and discussion
of time "Instants" may be preferable to Packet Loss, since these Instants
are indispensable to Rate-Derived Convergence Time. Also, there was
agreement to define a time period for evaluation of convergence stability
following the "Convergence Recovery Instant".

The meeting was reminded by an attendee that the BMWG should strive to
produce definitions that are understandable without pictures
(this is generally good advice).


-------------------------------------------------------------------------
5.  Terminology for Benchmarking Core Router Accelerated Life Testing.
      Changes to Term from 00 to 01. New Framework Draft 00.
      Application of Term and Framework to build forthcoming Methodology
      Issues? Comments? Concerns? (S. Poretsky et al.)

http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-term-01.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-framework-00.txt

The main comments on this topic sought better definitions of the terms and
benchmarks.  Unexpected Loss, a form of error that would trigger the
end of a test interval, needs to be clearly differentiated from other
losses that are expected (on the same interfaces).  There was a suggestion
to go beyond loss, and define errors in terms of packet delay and reordering.
 From a management interface perspective, very long delay on an SSH connection
is as bad as loss (can't manage the box).

The current benchmarks assess the relative performance
as a time interval of correct operation,
but have a Pass/Fail criteria as opposed to quantitative characterization.
The benchmarking methodology needs to characterize the DUT performance
along a dimension that allows easy comparison with other vendor's products,
so the approach should be to use stressful configurations that produce
errors in a short amount of time.  Discussion revealed that some participants
use this sort of testing in RFPs, and also to compare vendor's
products with under user-specific conditions,
and for this the terminology and framework may be enough info.
But there should be a very explicit connection
to the traditional black-box benchmarks added to these drafts.

-------------------------------------------------------------------------
6.  Revised Milestones. (Chairs)
The proposed milestones were presented without remarkable commentary.
Revise:
Dec 03  Resource Reservation Benchmarking Terminology to AD Review
Dec 03  Net Traffic Control Benchmarking Terminology to AD Review

Dec 03  Support EGP Convergence Benchmarking Terminology through AD Review
Dec 03  Support Multicast Benchmarking Methodology through AD Review
Mar 04  Support OSPF Convergence Benchmarking Drafts through AD Review

New:
Dec 03  IPsec Device Benchmarking Terminology to AD Review
Dec 04  Net Traffic Control Benchmarking Methodology to AD Review.
Dec 04  IPsec Device Benchmarking Methodology to AD Review
Dec 03  IGP/Data-Plane Terminology I-D to AD Review
Mar 04  IGP/Data-Plane Methodology and Applicability I-Ds to AD Review
Dec 03  Router Accelerated Test Terminology I-D to AD Review
Jul 04  Router Accelerated Test Method. and Applicability I-Ds to AD Review

Remove: (Pending FIB discussion/resolution)
Apr 03  Methodology for FIB related Router Performance Benchmarking to AD
review.
Jul 03  Resource Reservation Benchmarking Methodology to AD Review
Jul 03  Basic BGP Convergence Benchmarking Methodology to AD Review.

-------------------------------------------------------------------------
7.  New Work Proposals on Protection Switching Methodology (now there are 2)
      Should BMWG take a broad, or technology-specific approach to this work?
       - This is a placeholder for advanced discussion of this topic
       - There *will* be discussion on the list before the meeting to
         better articulate the questions posed to attendees and the list.

      Earlier proposal Automatic Protection Switching Benchmark Terminology
      The WG has discussed this item at several previous meetings.
      The Goal has been articulated as follows

      The objective of this effort is to produce a terminology and
      methodology set of drafts that specifies the performance
      benchmarking sub-IP layer resiliency and protection technologies.
      There is a common terminology draft and multiple methodology drafts
      for the technologies.  The methodology drafts will include (but not
      limited to) Automatic Protection Switching (APS) for SONET/SDH,
      Fast Reroute for Multi-Protocol Label Switching (MPLS), and
      Resilient Packet Ring (RPR) standardized in IEEE. (T.Kimura, J.Perser)

http://www.ietf.org/internet-drafts/draft-kimura-protection-term-02.txt


      New Proposal Draft on MPLS Protection Benchmarking Methodology
          The Goal has been articulated as follows
To develop a benchmarking methodology for MPLS protection mechanisms
including Headend Reroute, Standby LSP, Fast Reroute Detour Mode, and Fast
Reroute Bypass Mode.  Test cases will benchmark the DUT in all LSR roles
including Ingress, Midpoint, Egress, PLR, and Merge Node.  Test cases are
provided for Link and Node protection.  The most common causes of failover,
such as local administrative shutdown, local link failure, and remote link
failure are considered.    The Benchmark for each test case is calculated
from the measured packet loss during a failover event.  Benchmarks can be
used to compare failover performance of different Label-Switched Routers and
evaluate the different MPLS protection mechanisms.  The methodology uses
existing MPLS and MPLS protection mechanism terminology defined in current
IETF RFCs.  (S.Poretsky, et al.)

http://www.ietf.org/internet-drafts/draft-poretsky-mpls-protection-meth-01.txt

Discussion of these two new BMWG proposals vacillated
between the need for MPLS-specific failover benchmarks vs. generic IP
service protection provided by underlying transport mechanisms.
The two approaches appear to be solidifying as separate entities.
One participant identified the his need
to understand the performance of MPLS network elements in their native
mode of operation, at the Labeled Packet layer and in terms of LSPs or
VPNs.  Clearly, the original proposal/direction to characterize protection
effects at the IP layer retains merit, and the terminology draft now presents
a fairly advanced framework.

There is a clear need to recruit more participation in both these
protection-related work items (thus far discussion has been among the
authors and the WG chairs).  Thus the call(s) for support will include
requests for participants who will actively review the work,
having identified themselves as providing some valuable perspective
or expertise.

-------------------------------------------------------------------------
8. Trends in BMWG

The chairs managed to insert many comments on WG trends during the meeting,
including apparent lack of readership in certain areas beyond the author
list(s). There maybe a need for a change in the instructions for Last Calls,
where even agreeable readers must respond with some commentary, and
drafts do not progress until sufficient review has been completed.


_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg