[bmwg] Preliminary Minutes from IETF-64

[Al Morton <[e-mail]>]

BMWG,

Please provide any comments or clarifications by Nov 30, 2005.

To improve your reviewing experience, the slides from our
session (and others) are gathered here:
https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=64

Kevin/Al
BMWG Co-Chairs


<xmp>
Benchmarking Methodology WG (bmwg)

MONDAY, November 7, 2005

1510-1710 Afternoon Session II  (Oak)
=======================================

CHAIRS: Kevin Dubray <[e-mail]>
         Al Morton <[e-mail]>

MEETING MINUTES/REPORT:

** These Minutes are Preliminary, until December 9, 2005 **

Matt Zekauskas appeared in time to be graciously drafted as minute taker,
so his detailed notes, along with Kevin's Jabber Log,
http://www.xmpp.org/ietf-logs/[e-mail]/2005-11-07.html
were edited into the minutes by the Co-Chairs.
About 20 people attended the BMWG session.

0.  Agenda bashing (we may need to shuffle a few items)
     See 
https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=64

We bashed the agenda to move-up items that were somewhat controversial,
so that AD David Kessens could attend a conflicting meeting.

1.  Milestone Status (Chairs)
See Slide 6 in the Chairs slide deck for the list of proposed milestones.
Based on presumed misses, the Chairs propose to revise milestones by a year.
There has been clear progress made, but progress requires more than
work by the editors. Some of the 350-plus members of the WG-list
must help by providing reviews. Our revised milestones must represent
real commitments to complete the work. The meeting produced more changes and
revisions, and the new dates will posted for discussion.

The chairs noted that the FIB methodology draft has expired again
due to lack of interest/support, and they plan to remove this milestone
altogether, following notice of intent on the bmwg-list.

2.  Revised Charter discussion (Chairs)
See slides 7 through 9 in the chairs' deck.

The main discussion covered a point raised at the Paris IETF-63
regarding the scope of BMWG coverage.  The chairs proposed to
focus the WG effort "primarily" on internetworking technologies
developed by the IETF.  This clearly indicates our emphasis, but
also leaves some wiggle room so that we can consider exceptions
if they arise, because it is difficult to assess any technology
in isolation (IPsec is a classic example).

The charter discussion also covered points that are considered beyond
the scope of BMWG.  Compliance testing is definitely out of scope,
BMWG will assume compliance when a manufacturer claims it.
The consensus was that classic reliability benchmarks
are out-of scope, but also that the sub-IP restoration and current
stress testing are reasonably within the charter. There should also
be a de-emphasis on niche technologies; it is better to focus on the
"important" protocols that many operators use, and the level of
importance is a matter for the WG to determine through consensus.


3.  New Work Proposal (Chairs)

    Sub-IP Protection Mechanisms - (Poretsky)
	a. work item proposal
        We have drafts that implement a harmonized proposal, but there
        needs to be enthusiastic support to get this off the ground.
	b. progress achieved
http://www.ietf.org/internet-drafts/draft-poretsky-protection-term-00.txt
http://www.ietf.org/internet-drafts/draft-poretsky-mpls-protection-meth-04.txt

This work has a long history, but it has evolved into a harmonized
proposal, and grown with considerable interest of new participants.
It is very much like the current work on IGP-dataplane benchmarking,
and reflects the fact that operators are demanding better restoration
performance and are investigating sub-IP mechanisms.  Only 2
people in the room had read the drafts, so it was not possible to
develop consensus to adopt the drafts as a work item.  However, many
people *have* read these drafts and provided comments, it's just that
almost all of them are not present today, so there is active interest.
It may be helpful to contact other WG chairs to recruit reviewers and they
may join us in bmwg to work this topic: MPLS is a prime example.
Scott's next steps are to determine what other WG should be contacted
to review this work.  David Kessens reminded us that we need
more than interest to begin new work -- people must say that they
will read, review, and comment to make the result a WG product.

About 8 people were attending BMWG for the first time, and 2 of them
had engaged in this work item and read the drafts, so there has been
some successful recruiting already.

4.  Extended Working Group Status (Chairs)

BMWG will follow the new WG Chair Shepherding process for all
new publication submissions.  See:
draft-ietf-proto-wgchair-doc-shepherding-05.txt
This effectively means one WG chair takes on role of shepherd to
assure progress through AD and IESG review and the rest of the
publication process.

     Network-layer Traffic Control Mechanisms Terms (dsmterm)
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-dsmterm-11.txt
In his final read before shepherding this draft, Al found many typos that 
should
be corrected and a clarification that the "delay" mentioned in
several delay-related vectors is "Forwarding Delay".
This draft needs some minor Final Editing before submission to ADs for 
publication.

      Hash and Stuffing Draft
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-hash-stuffing-04.txt
The authors of this draft have requested a WGLC.  Version 04
has only minor changes, so now we need to test for consensus. We will
apply our active review process with the review templates, and need
review volunteers when starting Last Call.

     Benchmarking Terminology for Resource Reservation Capable Routers
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-benchres-term-06.txt
WGLC completed on this draft without comment, and the plan is to
submit this draft for publication.

The proposed work area on LDP convergence was briefly discussed
during the status discussion, and it was confirmed that this work
was on hold until the IGP-Dataplane work moves on.

5.  IGP Data plane convergence benchmark I-Ds (Poretsky)
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-08.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-08.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-08.txt
WGLC completed with all positive comments. Some editorial work
is needed to clean-up nits and typos before request for publication
(reference to Al's comments on the list).
Sue Hares has committed to review the drafts, so that should satisfy
our need for cross area review.  Scott implored the group to read this
now, so that the 09 versions can be submitted for publication.
There have been no technical issues raised on these drafts for months.

6.  Terms and Methods for Benchmarking IPsec Devices (Merike Kaeo)
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-ipsec-term-07.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-ipsec-meth-00.txt

WGLC was completed on term version 05 with comments, and version 07
reflects responses and other revisions stemming from development
of the methodology (available for the first time at this meeting).
Merike emphasized that there is lots of confusion about IPsec,
so benchmarking for IPsec needs to be extremely clear.

There have been many clarifications as the methodology was developed.
For example, new RFC 4021 lists the IKEv1 requirements.

The authors are asking reviewers **to identify any missing benchmarks**.
There should be consensus on test parameters, security context and frame 
formats.

Regarding the lack of coverage for IKEv2, it seemed to be best to complete
the work with IKEv1 and manual security associations, but Merike will
test this plan with a message to the list.  IKEv2 may be addressed
in a separate work item, as it may soon become widely deployed.


7.  Techniques for Benchmarking Core Router Accelerated Life Testing.
     (Poretsky)
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-term-07.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-04.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-ebgp-00.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-opsec-00.txt

Terminology & Methodology I-Ds were revised. The Methodology has been split
into 3 drafts (general, EBGP, & security). Editor wishes to test
readiness for WGLC on terms and general methods drafts and discuss the
next steps for the specific-topic methodologies.

The presentation included an example implementation of the terms
and methodology drafts, prompting very active discussion and detailed
feedback, including:

o  the benchmark Recovery Time would be better if measured with
sub-second resolution.

o  since the benchmarks span multiple dimensions (rate, latency,
lost sessions), it may be helpful to discuss the possibility that
comparisons may not yield superior performance for one device
in every category. However, the guidance must not go down the
slippery slope of interpretation or "compliance". The guidance
could address the general question, "what do we do with the benchmarks?"

o "degraded forwarding rate" is really a derived benchmark, not the
same as other rates, and should be named "forwarding rate degradation"

o  there were many suggestions and requests for more detailed
information on the start-up and general configurations. Specifically:
  - Stats enabled, but what options? (should be similar for both DUTs)
  - Telnet sessions in number enabled, but how much traffic? (Scott
reported this in his slides, but needs to require it in the methodology.)
  - Debugging - similar need for detail

o  Line card insertion and removal - agreed to be useful, but difficult
to develop a a "standard" procedure that would yield uniform results, and
physical insertions and removals would be complex to automate!

The chairs said that they believed that the terms and methodology
were ready to enter the WGLC process, following revisions to reflect
the comments received at this meeting.  The Last Call Process will
use the Active review templates to ensure detailed reviews and
hopefully accelerate the development of these drafts.  However,
the chairs asked that everyone defer development of the detailed
methodologies until we have progressed the general method and terms.

This topic was also discussed during the OPSEC working group meeting,
and additional comments may be captured there.


ACTION ITEMS:

+  Deal with charter and milestone updates

+  Restoration work proposal: looking for more volunteers and WGs to review

+  Merike will post message about IKEv1 vs IKEv2 in the IPsec documents.
    However, the WG has discussed this, should probably progress the current
    memos making it clear that they apply only to IKEv1.

+  WGLC on hash and stuffing

+  WGLC on accelerated stress term+meth (following revisions discussed)

+  Publication Requests on benchres, dsmterm w/rev,
    and igp-dataplane w/rev.

</xmp>


_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg