BMWG minutes

[Kevin Dubray <[e-mail]>]

Enclosed are the minutes of the BMWG meeting held in San Jose 
on 9 Dec 96 that I have submitted to the IETF secretariat.

-Kevin 

----- Begin Included Message -----

Minutes of the Benchmarking Methodology Workgroup
 
Reported by Kevin Dubray
 
Over 50 people signed the attendance list for this meeting.
This was an abbreviated meeting of the BMWG due to the extended
introductory IETF general meeting.
 
1. Agenda.

Kevin Dubray opened the meeting and offered the proposed agenda for 
this session of the BMWG.  The agenda was accepted as follows:

 -  Discuss recent modifications to the LAN Switch terminology
    draft.  Resolve any outstanding issues so that the draft can
    be forwarded for consideration as an informational RFC.
 
 -  Introduce the new cell/call benchmarking terminology draft.  
    Ascertain if direction is correct. Provide the editor preliminary
    feedback on initial drafting of terms.
 
 -  Introduce multicast benchmarking terminology draft.

 -  Review and update BMWG milestones.


2. LAN Switch terminology draft.

Dubray turned the floor over to the draft's editor, Bob Mandeville.
Mandeville led the discussion on the latest version of the switch
terminology draft, <draft-ietf-bmwg-lanswitch-01.txt>.  Bob began
the discussion by addressing areas of the text for which he had received
feedback.

Bob indicated that the most vociferous comments on the draft 
addressed the definition of "forward pressure".  Concern was expressed
that the concept, as presented, seemed to advocate this type of behavior
in order to mitigate frame loss in the face of congestion, specifically
collisions.  It was feared that, by extension, some interpretations of this
terminology may lead one to conclude that adherence to a networking standard
was optional.

Bob countered that was not his intent; rather, he was just trying to
define a behavior.  Doug Ruby questioned whether forward pressure was
actually measurable as defined.  Mandeville indicated that his experience
was that forward pressure was, indeed, measurable. He further thought the
concept was useful in drawing distinction between the behaviors of various 
switches.

A question was raised as to whether the pertinent components of the IEEE 
802.3 standard should be cited.  Scott Bradner went on record to say that 
he did not think so.  He believed that it was a good thing to keep a concept 
as generic as possible in order to retain the ability to use that concept in 
other scenarios. 

The group reached consensus in suggesting that the current wording
of "forward pressure" reflect that not adhering to an approved 
Standard is not an acceptable practice.
 
Another term for which Bob received feedback was "multidirectional traffic."  
Many thought that the term's name was not straightforward.  Scott Bradner 
commented that he has used the term "meshed" to refer to the traffic 
orientation suggested by multidirectional.  The group agreed that "meshed" 
should be used in lieu of term, "multidirectional."  Furthermore, Bradner 
pointed out, the concept of "fully meshed" should be dropped from the 
definition component.  Again, the group agreed.

Some confusion was expressed with regards to the term "burst" - specifically,
references to the notion of a "single frame burst."  Mandeville replied that
it is useful to describe a "packet in isolation."  He further commented that
addressing burst just extends the discussions presented in RFC 1242 and 
RFC 1944.

A discussion of "backpressure" followed.  Doug Ruby articulated that "jamming"
was just one technique of backpressure.  There are other techniques both
active and passive.  An example of active backpressure was the PAUSE flow 
control technique offered by IEEE 802.3x draft.  Further, Doug was not sure 
of the utility of the throughput metric in active backpressure scenarios.  

Doug advocated that test gear needs to responded to active backpressure 
scenarios.  Scott warned that it was his experience that
different test gear did not necessarily respond consistently to same
stimulus.  Scott further noted that reliance on any one test device may be
problematic.

Doug countered that unless the offered terminology reflects consideration 
for full duplex, flow control, etc., test gear vendors will not
build supporting tool sets.  Dubray seconded this thought, offering that it 
appeared that tool vendors sometimes consider BMWG work as functional 
specifications for their products.  Bradner asked Ruby if Doug was offering to 
supply wording to convey the ideas that he was advocating.  Both Ruby and 
Mandeville agreed to work towards this end.

On another topic, Mandeville raised the idea that it might be appropriate to
move some of terminology that Dubray has offered in his initial Multicast 
Terminology draft <draft-ietf-bmwg-mcast-00.txt> into the current LAN Switch
draft.  Bob indicated that Kevin was not against moving the generic terminology
into the switch draft.  Scott commented that generic concepts didn't really 
fit into a multicast specific discussion, anyway.  Kevin pointed out that his 
goal was not to introduce a battery of unrelated terminology.  Rather, in the 
course of discussing multicast related terminology, it became evident that 
concepts alluded to in previous BMWG work were either undefined or implied.  
Kevin didn't particularly care where the items were concretely defined per se; 
however, inclusion in the multicast draft seemed most expedient and helped 
the multicast discussion.  

He further went on to say that the tactic wasn't unprecedented, citing the 
previously described term of "meshed". "Meshed" traffic went beyond the 
context of switch testing, but it was being defined in the switch draft.  
Someone offered that the power of the BMWG effort did not rest in any one 
document, but the collection of documents taken as a whole.  The group agreed.

Some of the terms from the Multicast Terminology Draft that Bob thought would
have use in the LAN Switch draft were Device Under Test (DUT) and System
Under Test (SUT).  Scott reflected that DUT was fast becoming an  
uninteresting term.  Dubray countered that the notion of a single device
interacting only with the test tools was important. Moreover, it was a
simple concept from which other relationships could be built.  Again, the group
agreed.

Bob furthered the discussion on the LAN switch draft by contrasting how
the LAN switch draft defined the concepts of "nominal load" and "real load" 
while the Multicast Draft conveyed the notion of "target rate" and "offered
rate".  A question from the floor queried why the need for distinction a
between nominal/target versus real/offered.  Kevin offered that it was his 
experience that in the course of testing, there were occasions when the
test tool may be unable to generate frames at the requested rate.  The 
reasons for this may be DUT related (e.g., collisions as a function of
flooding conditions) or tool related (e.g. the tool unable to utilize the
maximum usable bandwidth of medium).  It is useful, then, to draw 
distinction between these two datapoints.  After some small discussion
the group agreed to use the terms "intended load" and "offered load".

A short discussion followed on the notion of "speed" as defined in 
the Switch draft versus "forwarding rate" as presented the Multicast draft.
Bob stated that speed only considered the forwarding ability of at the points
of egress from the tested system.  Bob continued that the importance of 
"speed" was in the metric's ability to describe a system's behavior in the 
face of congestion control. Dubray offered that the forwarding rate
described the egress component as a function of the ingress component.  In
other words, forwarding rate is the observed rate at which a tested device
forwards frames for a specific offered load.

A question from the floor pondered whether "maximum speed" and "maximum
forwarding rate" were equivalent.  Dubray thought not.  Maximum speed
conveyed the maximum number of frames a DUT could forward in a fixed time
frame (a second), without consideration for the offered load. This differs 
from the presented definition of maximum forwarding rate, which described 
the DUT's ability to forward frames in response to the maximum, tested 
offered load.  Many thought this not to be intuitive.  Scott echoed that 
most people seem to consider "maximum frame rate" to be the "largest" 
forwarding rate taken from a set of forwarding rate measurements.  Dubray 
agreed the title of the term was not intuitive as presented.  However, he 
continued, it was his experience that comparisons between the "largest" 
forwarding rate and the forwarding rate measured at the highest offered load 
gave significant insights into a device's behavior.  Scott concurred and 
suggested the benchmark be given the name "forwarding rate at maximum offered 
load." The group thought this to be very intuitive.  Dubray stated that this 
was consistent with email that he had received on the topic.

Dubray interrupted the discussion due to time constraints and said that he
and Bob would discuss further consolidation of terms off-line.  In assessing
the overall shape of the LAN Switch draft, the group concluded that with the 
discussed enhancements and other minor modifications folded into the draft,
there were no major blocking factors to the document's advancement.  Bob 
indicated that he would issue a revised draft as soon as possible.

3. Cell/Call Terminology Draft.

Dubray announced that the draft's author, Robert Craig, was 
unable to attend the BMWG session.  In his place, Bob Mandeville
had cheerfully volunteered to lead the discussion on this initial draft.

Bob immediately began a rigorous discussion of the draft, starting with
section 3.1, Virtual Circuits.  Questions were raised as to the scope of the
call specific items, such as Call Setup Time and Call Setup Rate.  Were  
these benchmarks end-to-end measurements or network element-to-network element 
measurements?

Scott chimed in that one of the questions that he is often asked to answer
is how to qualify the overhead of a tested system in that system's delivery 
of a requested function.  Scott communicated that defining a generic 
system-level response benchmark may have merit. Such a benchmark would have 
utility, whether it was used in measuring call setup up time or an HTTP 
transfer.  Some questioned whether such a generic benchmark would capture 
the specific nature of the transaction being measured.  More pointedly, 
another person asked about the nature of the question that the draft was
trying to answer.

Dubray halted the discussion to focus on the group on item 2 of the
agenda: ascertain whether the direction of the current draft was correct
with respect to the previously agreed upon workplan.  That workplan, Dubray
reminded, was to develop a set of metrics and terminology that helped
characterize a system of devices that forwarded frames from "legacy"
based technologies via a cell-based/connection-oriented infrastructure.
Dubray added that he thought the scope of this effort was to provide the
terminology surrounding a test scenario not unlike that conveyed in section 
19 of RFC 1944. The focus of the exercise was not to concentrate on a 
particular network element, but rather a system of devices comprised, 
perhaps, of edge devices and switches.

The group thought that in its current form the draft was slightly off-mark by
focusing on specific network elements.  However, there was an overwhelming
agreement to continue the draft in its current direction.  The discussion
that followed was targeted in giving the draft's editor general feedback.

The most prevalent comments on the draft reflected the fact that the draft 
introduced terms that may be outside the scope of benchmarks and related 
terminology.  For example, terms like "Buffering Strategy" or "Queuing 
Strategy," maybe outside the intended scope of the draft as they may be more 
architectural in nature.  Dubray noted that email by Jim McQuaid on the 
BMWG mailing list reflected this fact and Robert agreed to drop terms of 
this type.

On the proposed term "non-blocking factor," most thought that the concept
was interesting, but they also thought the formal definition was weak and
needed clarification.  Mike O'Dell thought it appropriate for the draft 
to consider the notion of "blocking probability."

With regards to section 3.4 (Buffering), someone recalled that there was 
some work done in this area by the ATM Forum, and that it may be prudent
to survey that work.

On the topic of forwarding measurements, Mike O'Dell thought it would 
be helpful to introduce some idea of "load dependencies".  Mike went on
to say that he thought it would be very useful to determine how invasive 
are the changes to the forwarding path on forwarding performance.

In response to item 3.2.1 and 3.2.2 (Packet disassembly/reassembly time,
peak and sustained, respectively), some conveyed that it would be impossible
to measure the SAR function explicitly as a black box style test. 

The group generally thought section 3.7.1, traffic management, 
needed reworking.  It wasn't clear what was being tested, policing or 
admission control.  

Scott Bradner offered a suggestion that the editor contact Dr. Raj Jain
of Ohio State and solicit feedback and suggestions. 

With that, Dubray closed the discussion on the cell/call terminology draft 
noting that the minutes would be forwarded to the editor. He thanked 
Bob Mandeville for being the discussion moderator. He also noted that due to
the abbreviated session, there would not be an opportunity to discuss the
Multicast Terminology draft beyond what had previously been discussed.  He
encouraged people to read the draft and comment via the BMWG mailing list.

Goals for the next BMWG session in Memphis.

1. Edit LAN switch draft to reflect the agreed upon modifications.  Distribute
   document to mailing list for comment.  If no blocking issues are
   identified, ascertain consensus on whether to forward draft for 
   consideration for RFC status.

2. Continue progress on the Cell/Call Terminology Draft:  address issues
   and concerns raised during the BMWG session; revise and reissue draft and 
   resubmit for comment. 

3. Review and comment on Multicast Benchmarking Terminology Draft; revise
   and reissue draft as necessary.


----- End Included Message -----