[bmwg] Draft minutes for BMWG session at IETF-61

[Al Morton <[e-mail]>]

BMWG,

The draft meeting notes follow.  Please send
any comments by Tuesday, December 7, 2004.

thanks,
Kevin/Al


Benchmarking Methodology WG (bmwg)

Thursday, November 11 at 0900-1130
=====================================

CHAIRS: Kevin Dubray <[e-mail]>
	  Al Morton <[e-mail]>

This meeting report was prepared by the chairs, based on
detailed minutes collected by Phil Chimento as official
note taker. There were 30 people in attendance,
and Al Morton chaired the meeting.

AGENDA:

1. Working Group Status
-----------------------
Co-chair Al Morton highlighted areas of progress and areas
needing attention.  The OSPF Convergence Benchmarking Drafts
and EGP convergence terminology have been approved, and are now
in the RFC Editor's queue. Drafts in Last Call not discussed below
include the Resource Reservation Terminology (new version expected
soon for Last Call) and the Diffserv Traffic Control Terminology
(Jeff Dunn and Cynthia Martin volunteered to assume the editor
role with Scott Poretsky, since progress had stalled).
The Howard Berkowitz indicated that development on the EGP methodology
was resuming with input from Sue Hares (who has IRTF work
in-progress on BGP convergence). The summary of new work items
indicated that Hash and Stuffing proposal closed without objections,
and that the current plan was for the Network Convergence Draft to
proceed as an individual submission. Thomas Eriksson reminded the group
of his LDP convergence proposal, and that a new draft was available.

During the meeting, there was a short discussion on future scheduling.
Cynthia Martin pointed out the need for work between meetings, and suggested
another 2.5 hour session at Minneapolis. In addition, we need to avoid
overlap with the IPv6 session in the future, and there was a request
not to meet at 0900 Monday morning.

2. Revised Milestones
---------------------
The milestones have been revised on BMWG's charter page. The chairs
will work with AD David Kessens to add a milestone for new work on
Address Hash and Bit Stuffing.

3. IPSec Terminology Update (M.Kaeo)
------------------------------------
Editors were distracted during the development and missed the deadline, but
a new version of IPSec terminology will be done soon, followed by the initial
methodology.  Merike said they would like to see some review of the
methodology before a WGLC on the terminology draft.

4. IGP Data plane convergence benchmark I-Ds (S.Poretsky)
---------------------------------------------------------
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-04.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-04.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-04.txt
Scott described changes from 03 to 04 to address comments raised
at the last meeting. The method should be agnostic to layers below IP, so the
preference for SONET was removed. There were several recent list-comments 
based on
some early implementation experience, and other comments seek to clarify
definitions and the status of equations 1 and 2. Al Morton commented on
the methodology draft section 3.2.3, where we should state that forwarding
rate is measured on the next-best interface or the restored interface.
Scott will incorporate these comments in the next revision for
another WGLC.

5. Techniques for Benchmarking Core Router Accelerated Life Testing (S. 
Poretsky)
---------------------------------------------------------------------------------
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-term-04.txt
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-01.txt
Scott has added new test cases on new EBGP peer, change in BGP Policy,
and specified SYN Flood as the DOS attack in the methodology.
Jeff Dunn suggested a new test case with route flap-dampening on.
Howard Berkowitz asked if there was an implicit assumption that
peers have the same BGP implementation. There are radical
differences in time of convergence in different implementations, where data
structures in the routers are very different and affect the time.
This should be noted in the methodology.
George Jones asked if this document had cases where you throw
tons of traffic at the device and see whether authentication and other
management functions still work?  Scott answered yes, and offered to work
on these. Al suggested that the Start-up, Instability, and Recovery Phase
slide would be useful in the Draft.  Scott said he would try to add it.
Also, with the methodology still in development, it would be best to
wait on a last call for the terminology draft until there is more stability.


6. New Work on Hash and Stuffing (T.Player)
-------------------------------------------
http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-hash-stuffing-01.txt
Timmons gave a brief overview of changes in the 01 version, including a
definition of random in bit patterns.  The other changes avoid accidental
use of multicast addresses, and include considerations for MPLS Labels.
This draft did not have much readership this time, and Al suggested that
editors post interim drafts and avoid the avalanche just before the meeting.


7. Resumption of work on the FIB methodology with new editors (J.Dunn)
----------------------------------------------------------------------
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-fib-meth-02.txt
(The tombstone may still be here, watch for updates)
Jeff described the changes in this new, heavily revised version.
Co-editor Cindy Martin added that this was an expired draft.
Assumptions were being made in the document about the interactions
between the FIB and RIB which were not true. There was a short
discussion of vendors building to the tests. Scott Bradner commented that
it is almost impossible to avoid people building to the
test, so you have to construct the tests so that building to the test
doesn't matter. Bradner was also concerned about the idea of mimicking
traffic, but you can mimic topology. Jeff asked for more input on test
cases, and posited a RIB to FIB convergence ID.  Scott Poretsky reminded
that we already have control and dataplane convergence work to cover this,
and we can't approach "white box" measurements (seconded by others).
Howard Berkowitz pointed out that "graceful" recovery mechanisms
mean that routers operate for some time with a corrupted FIB,
and the draft should take this into account (Jeff agreed).
Al Morton asked to consider revising the Terminology RFC, since the
new methodology intends to include the effects of Route Aggregation
and the current terminology indicates that aggregation would be avoided.


8. IPv6 Benchmarking Introduction (J.Dunn)
------------------------------------------
Jeff gave a short summary of IPv6 concepts and testing requirements.
There was a discussion of the charter, since it doesn't mention IPv6
explicitly. Scott Bradner pointed out that BMWG's original charter
preceded IPv6 by about 5 years, and yet there are 10 BMWG I-Ds that
mention IPv6. In IETF, charters are contracts = what a WG will work on.
David Kessens supported putting IPv6 explicitly in the charter, so it
will encourage more people to work on IPv6.
Scott Bradner investigated the nature of Dual Stack testing with
several questions, and it was resolved to test with both IPv4 and v6
in a single configuration at the same time.  Scott also suggested that
the most efficient way to add IPv6 to the existing work was to
write a document with the new packet types, and then say in the
document "add these to the procedures in 2544". Update but don't replace.
A discussion of the hop-by-hop and extension headers followed.
In most circumstances, only the hop-by-hop header would be relevant,
because the extension header is only processed by the destination host.
Jeff is looking for folks to help out with this effort.


9. New Work on WLAN Benchmarking Methodology (S.Bradner, D.Stanley)
----------------------------------------------------------------
Related Draft:
http://www.ietf.org/internet-drafts/draft-alexander-wlan-meth-00.txt

Tom Alexander and Scott Bradner prepared an methodology draft for 802.11
devices and systems, and asked the bmwg-list if there was interest in this 
work.
Scott began his presentation saying that substantial interest had been
expressed, but that he was not proposing this draft become a BMWG work
item at this time. There is potential overlap with the IEEE 802.11 Task Group T
(TG T) on Wireless Performance Prediction. However, Tom Alexander is a 
member of
the task group, and he contacted Scott with the plan to prepare an Internet-
Draft for BMWG.

Scott gave a brief overview of the draft, which he characterized as a first
cut.  This methodology builds on the existing RFCs for Network Interconnection
Devices and LAN Switching Devices (RFCs 1242, 2544, 2285, and 2889).
There are three test set-ups, including a set-up for wireless clients.
One of the goals of the test conditions is to minimize the radio dependencies.
Scott closed his presentation (in much less than 20 minutes) by offering
three possibilities: The work could be done in IEEE 802.11, or as a
cooperative effort between 802.11 and BMWG, or entirely in BMWG.

Scott introduced Dorothy Stanley as the IEEE 802.11 Liaison to the IETF.
Dorothy presented a slide summarizing all the 802.11 task groups, and
highlighted (see slides). Task Group T is the IEEE 802.11 group that
has been formed to look at performance. They will produce a recommended
practice. Al Morton asked about the scope of the Wireless Performance
Prediction Task Group, and Dorothy responded by displaying a passage
from the Draft IEEE Project Authorization Request:
"The scope of the project is to provide a set of performance metrics,
measurement methodologies, and test conditions to enable measuring and
predicting the performance of 802.11 WLAN devices and networks at
the component and application level."
In response to David Newman's questions, it was clarified that component
level refers to the radio equipment aspects, and that the application level
refers to user applications - the IP layer was not specifically called out.
(Document 1157 outlines TG T work, and includes use of user opinion models
to interpret results, including ITU-T Rec. G.107 for voice quality).

David pointed out that there was no mention of mapping performance to
levels of traffic, as is done in BMWG. The scope did not seem to address
re-association time measurement.  Dorothy indicated that the "R" Task Group
had defined this time interval (from last packet sent or received on the
old access point to the first packet on the new access point.
Dorothy supplied a URL for all IEEE 802.11 documents:
http://www.802wirelessworld.com/index.jsp
(you may need to register to obtain documents,
the guest login appears restricted).

David Kessens indicated his hesitancy about taking up this document in IETF.
Lots of the measurements depend on the physical layer, and you may be
mostly measuring the radio. However, the capacity aspects are not radio,
and they are useful information. It is not clear whether the entire draft is
within the BMWG scope, but some is.

Bernard Aboba, IEEE 802 Liaison to the IETF,
quickly showed 3 slides to indicate some of the challenges
present in the context of 802.11 measurements and benchmarking.
Implementation of rate adaptation and roaming methods are key points.
IEEE has not defined how to do rate adaptation or even what the results 
should be.
Rate adaptation derives feedback from Frame loss. Bernard's first slide showed
RSSI vs Delivery Ratio, derived from an MIT study. There was an enormous amount
of scatter: Packet Loss Ratio can be anything from 20% to 100% under
some conditions. They attribute a lot of this variability to multipath.
As a result, rate adaptation can be very noisy, with lots of weird behavior.
The theoretical curves are very different. It was noted that the draft
addresses tunneling from one Access Point to another.

Bernard supplied the following URLs with more background information:
The slides on FER vs. S/N theory and observation are from "Link-Level
Measurements from an 802.11b Mesh Network" by Aguayo,
Bicket, Biswas and Morris of MIT:
http://www.pdos.lcs.mit.edu/roofnet/sigcomm-talk.ppt
http://www.pdos.lcs.mit.edu/roofnet/ExOR-HotNets.ppt
The best citation on roaming interval theory and measurement are the
following presentations to 802.11r by Areg Alimian and Bernard Aboba:
http://www.drizzle.com/~aboba/IEEE/11-04-0377-01-frfh-analysis-roaming-techniques.ppt
http://www.drizzle.com/~aboba/IEEE/11-04-0378-00-roaming-intervals-measurements.ppt

Scott added that repeatability is extremely important in Benchmarking.
He had run tests that were repeatable to within about 1% a year later,
but that buffer capacity testing in RFC 1944 that was not very useful
(with 17% variability in measurements, it's difficult to compare devices).

When asked his opinion on the best venue for the work, Bernard gave three
key considerations: Where is the demonstrated expertise?
Where are the people who will do the work? Where is the best community to
support the work?  Scott related Tom Alexander's opinion that BMWG
expertise was needed to complete the work, and they tried to match
the current draft to BMWG's charter as much as possible. It may be
appropriate to split work, appoint liaisons, etc. Overlap can be avoided by
working-out the assignments among the people directly involved.

David Newman moved the discussion toward the issues that are usually
addressed by BMWG, at IP level and above. 802.11 roaming events can cause
a transport gap of 10 to 15 seconds.  Part of this is TCP wake-up.
David added that it would be better to use IP packet length in the draft.
Also, the existing terms of Throughput and Latency (at Throughput Load)
were not so meaningful in the lossy wireless context. Bit error ratio
testing was discussed briefly, but thought to be out of scope.

Scott concluded that we need to work out a division of labor on this topic,
and we will hear from the IEEE 802.11 TG T after their meeting (week of 11/15).
The chairman/Al asked that both Liaisons reflect the comments in this session
back to their respective committees. Specifically, he raised
the question whether any committee can produce repeatable benchmarks
in this area.  Scott said that one goal of the current draft was to
minimize the radio-specific aspects of the testing, and hence minimize
the variability of the results.  The Signal to Noise Ratio Section (3.5.9)
is primarily where the topic is addressed (there are
other key points throughout Section 3.5 - Test Conditions).
The BMWG would like to know if the TG T members agree that
the repeatability goal was achieved in Tom & Scott's draft,
or what modifications/additional stipulations would be necessary
to achieve it.

Bernard added that a review of the draft could be requested.
A complete review is not asked, unless the TG T committee
decides that BMWG is the right place for some aspects of the work.


10. "Old" Work Proposal on Protection Switching Methodology (S.Poretsky)
------------------------------------------------------------------------
Drafts related to this proposal:
Draft on MPLS Protection Benchmarking Methodology
http://www.ietf.org/internet-drafts/draft-poretsky-mpls-protection-meth-03.txt
Automatic Protection Switching Benchmark Terminology, Expired, see
http://www.watersprings.org/pub/id/draft-kimura-protection-term-02.txt

The WG has discussed this item at several previous meetings.
Generic Sub-IP and MPLS proposals can now be re-combined into one
using a common terminology. Methodology was submitted without terminology
and used MPLS terminology. The terminology document was submitted as a
silver bullet covering all types of protection (including SONET, etc).
Now there will be a single terminology draft, and a methodology for
MPLS protection benchmarking.  Scott described the recent spike of
interest and review comments based on trial implementation.
All the discursion should be moved to the bmwg list.

About 6 people had read the drafts. Al Morton asked the folks who have read 
the drafts
to hum if they supported the work (Good hum), and no readers opposed 
(silence).
Al also asked if there was any enthusiasm among non-readers to review the
drafts, based on the presentation (no response). There was clear support from
folks who have knowledge. Updates will be submitted in the coming weeks.
We will wait to look at revised drafts and circulate a work proposal
on the list. Scott's plan is to Update Terminology with new terms
and Update Methodology with new terms and comments.

11. Quick Wrap-up
-----------------
Action Items from this meeting:
  - Note to Internet-Drafts to reactivate the FIB draft.
  - Charter update to include IPv6
  - Milestone addition for Hash and Stuffing
  - Looking for updates on almost every draft
  - Awaiting results of the IEEE 802.11 TG T discussions
  - New work Proposal on Protection Switching on the list


Mailing list archive:
---------------------
http://www.ietf.org/mail-archive/working-groups/bmwg/current/

Current Status of WG drafts:
----------------------------
WG Last Call
<draft-ietf-bmwg-fib-meth-02.txt>,  New Editors, new draft @ IETF-61
<draft-ietf-bmwg-igp-dataplane-conv-term-04.txt> Revised on WG input
<draft-ietf-bmwg-igp-dataplane-conv-meth-04.txt> Revised
<draft-ietf-bmwg-igp-dataplane-conv-app-04.txt> Revised
<draft-ietf-bmwg-dsmterm-09.txt>, Call ended w/editorials, EXPIRED, new editor

I-Ds
<draft-ietf-bmwg-ipsec-term-03.txt>, draft 7/04, new draft very soon, LC
<draft-ietf-bmwg-benchres-term-04.txt>, (Exp) new draft soon, LC
<draft-ietf-bmwg-acc-bench-term-04.txt> Revised
<draft-ietf-bmwg-acc-bench-meth-01.txt> Revised on WG input
<draft-ietf-bmwg-hash-stuffing-01.txt> Revised on WG input

Expired BMWG I-Ds
<draft-ietf-bmwg-bgpbas-01.txt>, Authors re-assembling
<draft-ietf-bmwg-benchres-method-00.txt> Pending term prog.

New Work proposals.
Hash and Stuffing - need to add to charter
WLAN benchmarking methodology - deferred to IEEE 802.11T
Protection  benchmarking - One Proposal - Float Proposal on List
Considerations for Measuring Network Convergence, Individual

RFC Editor Queue --
<draft-ietf-bmwg-conterm-06.txt>
<draft-ietf-bmwg-ospfconv-term-10.txt>
<draft-ietf-bmwg-ospfconv-intraarea-10.txt>
<draft-ietf-bmwg-ospfconv-applicability-07.txt>

New RFC 3918   was <draft-ietf-bmwg-mcastm-14.txt>



_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg