Re: [bmwg] Comments for OSPF Convergence Benchmarking Terminology

Scott Poretsky wrote:
> Kevin,
> 
> I thought BMWG drafts are not permitted to have White Box test 
> coverage.  Is this correct? 

The focus of BMWG is providing tests that have general applicability.
Tests relying on internal (vendor-specific) mechanisms may be
dissimlar in what they report or how they report it. This
variance is generally the cause for us to steer clear of
"white box" measurements..

That said, if my read of <draft-ietf-bmwg-ospfconv-applicability-01>
is on, use of the internal mechanism, while not good for
vendor comparison, may be good for building understanding
of an implementation. For example, knowing how a device's CLI-reported
telmetry varies from the externally measured results may be
valuable insight.  (And insight is what we're all about here
in the BMWG. :-))

A suggestion might be to relegate all internal measurements and
their methodologies to an appendix with all the appropriate
warnings and caveats.  (Not "official" BMWG metrics; applicable to
a single implementation; use to reconcile internal measurements to 
external measurements, not to be cited for multivendor comparison,
blah, blah, blah.)

  I also thought that existing terminology
> should either be excluded or referenced in the Existing Terms section.  
> Is this correct?

Historically, "existing terms" was used to draw relevance to
other BMWG docs to ensure consistency.  This section, alas, has
not been used consistantly.

But to Ross' point: drawing relevance of the terms to use in
associated benchmarks is important.

Let's look at an example: Hello Interval.  Here the OSPF convergence
terminology I-D doesn't recast otherwise contradict references
in RFC 2328, a standard. So, I don't really have issue with
it's inclusion in the I-D.  (In point of fact, I find it useful
to have it inline with the draft.)  The discussion on timers
is important to this class of benchmarks.  Calling out related timers
is appropriate. (And required, if you ask me.)

For the lawyers, an alternativate might be:

      Hello Interval-

      Definition: see RFC 2328, sec. 9, sec. 9.5, sec. 10.

and move everything else to the "Discussion" section.

FWIW,
Kevin

> 
> Scott
> 
> At 06:24 PM 3/10/03 -0500, Russ White wrote:
> 
>> > Internal Measurements - This is a redefinition of White Box 
>> measurement.
>> > Recommend it be removed.
>>
>> The definition is not the major point of this defintion, but rather the
>> discussion about internal measurements, that they can impact the
>> performance of the box being measured, etc. I don't know if there is a
>> "generic" draft which covers these sorts of things, or if one is planned,
>> but it could go there just as well as here. In the meantime, it seems 
>> like
>> a useful discussion to have on paper.
>>
>> > External Measurements - This is a redefinition of Black Box 
>> measurement.
>> > Recommend it be removed.  In the Discussion section for External
>> > Measurements it states:  "when a downstream device receives complete
>> > routing information from the DUT, it can be inferred that the DUT has
>> > transmitted all of the routing information available". How can tests be
>> > built so that it is not "inferred", but instead "known"?
>>
>> You can't do more than infer, in most cases. You can state that it is
>> "known" that the DUT has finished sending information, but you don't know
>> if you are going to, for instance, drop the last ack, and force the 
>> DUT to
>> retransmit.
>>
>> > Multi-Device Measurement - This is a definition describing a test with
>> > multiple DUTs.  This term serves no purpose.  In fact, it does not even
>> > appear in the companion methodology.  Recommend it be removed.
>>
>> Actually, in our definition, a multi-device test requires measurement of
>> the test results on several devices, not necessarily several DUTs. For
>> instance, you may need to measure the timestamp on both the route 
>> generator
>> and the DUT, which means it's a multi-device mesasurement (not a
>> multi-device test, which would imply something different). I don't argue
>> that we didn't use the term in the methodology, but we certainly 
>> described
>> some multi-device measurements in some of the tests.
>>
>> Again, the discussion is important here, in how some of these issues
>> surrounding multi-device measurements.
>>
>> > Point-to-Point Link - Term was already defined by the Point-to-Point
>> > Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for
>> > the Transmission of Multi-Protocol Datagrams over Point-to-Point Links.
>> > Recommend it be removed.
>>
>> First, the concept of a point-to-point link encompasses more link types
>> that the PPP working group covers. For instance, there is a draft in the
>> IS-IS and OSPF working groups which discusses point-to-point ethernet
>> links, and how these protocols should treat these links different than a
>> normal ethernet link. In this case, I'm not certain the duplicate 
>> defintion
>> is a bad thing, an the discussion about convergence times in OSPF 
>> vis-a-vis
>> point-to-point links needs a home.
>>
>> > Broadcast Link - While this term is not defined, but frequently 
>> used, in
>> > IETF documents, it is a well-known term defined in IEEE 802 documents.
>> > Recommend it be removed.
>>
>> Note the definition is in terms of OSPF, and the benchmark draft. 
>> Again, it
>> may be a commonly known term, but the concept of a DR being elected on 
>> the
>> link is something which is specific to OSPF.
>>
>> > SPF Time - This is a white box parameter that is not externally
>> > observable.  RFC 2328 already defines SPF.  Recommend it be removed.
>>
>> There are tests included which discuss how to measure SPF time from 
>> outside
>> the box, and inside the box. I'm fairly certain the methodology draft
>> includes white box measurements, so I'm not certain why white box
>> measurements shouldn't be included in the terminology draft?
>>
>> > Hello Interval - This is a term already defined in RFC 2328, OSPFv2,
>> > Section 9, The Interface Data Structure and its use described in detail
>> > RFC 2328, OSPFv2, in Section 9.5, Sending Hello Packets and Section 10
>> > Neighbor Data Structure.  Recommend it be removed.
>>
>> The discussion here is just as important as the definition, and 
>> applies to
>> all tests discussed in the methodology draft. Suggestions on where 
>> else we
>> would put this discussion?
>>
>> > Router Dead Interval - This is a term already defined in RFC 2328,
>> > OSPFv2, Section 9, The Interface Data Structure and its use 
>> described in
>> > detail RFC 2328, OSPFv2, Section 10 Neighbor Data Structure.  Recommend
>> > it be removed.
>>
>> See above, on hello interval.
>>
>> > Incremental-SPF - This is a white box component of OSPF. Implementation
>> > of this feature is optional. Superior Convergence times can be obtained
>> > without Incremental-SPF (as two vendors have demonstrated).  Black box
>> > measurements cannot indicate if Incremental-SPF is internally 
>> implemented
>> > in the DUT.  RFC 2328 makes one reference to Incremental-SPF as an
>> > example algorithm to improve SPF run-time.  It does not require it nor
>> > indicate it to be the _most_ efficient algorithm.  If we were to
>> > standardize internal SPF execution algorithms then we should also 
>> include
>> > PRC, RTC, and others.  There would be no reason to choose just
>> > Incremental-SPF.  Recommend it be removed.
>>
>> I don't see it used in either draft, so this is fine.
>>
>> > Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This
>> > is a partial measurement of router components that contribute to
>> > Convergence time.  Time to update FIB is certainly a missing component.
>> > Also, network operators speak of "Convergence" as the time it takes for
>> > all nodes in the network to converge.  I propose that this term be
>> > renamed "Protocol Convergence" or "Control-Plane Convergence".
>>
>> Either way is fine with me.... Of course, this definition draft only sets
>> the meaning of words for the accompanying methods and applicability 
>> drafts,
>> and not in general. Others have defined convergence differently, so.... I
>> suppose that we, as a working group, can go one of two directions on 
>> this:
>>
>> -- We come up with a new name for convergence, prepending it with some
>> other word, to imply different things in different places. This could
>> probably lead to large and protracted arguments, and perhaps
>> overspecification of the meaning of words.
>>
>> (for instance, we've already added "SR" to the front of convergence for a
>> similar objection, so now we'll have:
>>
>> -- SR-CP-Convergence, Single Router Control Plane
>> -- N-CP-Convergence, Network Control Plane
>> -- SR-DP-Convergence, Single Router Data Plane
>> -- N-DP-Convergence, Network Data Plane
>>
>> as a base set of variants, with, I assume, many many more to come).
>>
>> -- We allow each draft to use the word "convergence" within it's bounding
>> limits (the dictionary definition), and just make certain that each draft
>> discusses its definition of convergence at some point, to clarify for the
>> readers of that draft. always remembering to have a flexible mind, and 
>> seek
>> out the meaning of convergence before reading any given BMWG draft.
>>
>> I'll defer to the working group for that discussion and decsion, though I
>> prefer the latter.
>>
>> > 2. Measuring Traffic Convergence Section
>> > Three measurements are listed.  Here are the concerns:
>> > A. LSA Propagation
>> > B. SPF Execution - This is a White Box Measurement
>>
>> A test is included in the methods draft, so I'm not certain why this
>> shouldn't be here?
>>
>> > C. RIB Update - How can this be measured in isolation from 1 and 2
>> > without it being a white-box test?
>>
>> We don't include a test for this, so whether it is white box or black box
>> makes no difference? It's just noted as a part of overal convergence 
>> times,
>> to be complete in the discussion of the elements of convergence time.
>>
>> > 3. Type of Network Events Section
>> > Next-Hop change (e.g. due to metric change) is a unique event 
>> because no
>> > packet loss is expected, so this should also be listed.  The other 
>> events
>> > listed - Link Up/Down, Session Initialization, and Adjacency Down
>> > produce packet loss.
>>
>> We didn't discuss packet loss in any section, since we are concerned with
>> contorl plane convergence, rather than data plane convergence (?). We can
>> add next-hop change, although I'm not certain we base any tests on it, 
>> and
>> making this list exhasutive would, well, unless we want to do that as a
>> generic document, we probably don't want to go there. :-)
>>
>> > 4. LSA and Destination Mix
>> > No new terminology.  Methodology related information.  Recommend 
>> move to
>> > methodology document.
>>
>> This could also be moved to the applicability document, but we tried to
>> move these sorts of discussions outside either one of those docs, since
>> they didn't seem to match the flow of the documents themselves. I don't
>> know, actually, where we would put them in the methdology draft--it would
>> have to either go in the front, in a general considerations section of 
>> some
>> type, or it would have to go in the individual tests, repeated as needed,
>> or at the end, where it likely wouldn't get read.
>>
>> I'll defer to the working group on this one--I see no compelling 
>> reason to
>> move them or to leave them.
>>
>> > 5. Tree Shape and SPF algorithm
>> > These are White Box items.  Neither is externally observable.  Tree 
>> shape
>> > is already described in RFC 2328.  I recommend that this section be
>> > removed.  It could be added as an appendix to the methodology 
>> document, but
>> > that is not worth it since it is already covered in RFC 2328.
>>
>> First, I'm not certain where we are defining these drafts as black box
>> only? We refer to a paper on black box measurements, but I don't think
>> this methodology draft is confined to black box measurements. There are
>> specific warnings in various discussions about using white box
>> measurements, and the problems you can have with them.
>>
>> Second, the concept of tree depth and tree breadth might be considered in
>> RFC2328, but I'm not certain it's not worth discussing it again, from the
>> point of view of the impact the shape of the topology you feed to the
>> routers can have on the test results.
>>
>> As for where it should be, again, I see no reason to move it or not move
>> it.... I'll defer to the chair and the rest of the working group on this
>> one.
>>
>> :-)
>>
>> Russ
>>
>>
>> -- 
>> riw@cisco.com CCIE <>< Grace Alone
>> _______________________________________________
>> bmwg mailing list
>> bmwg@ietf.org
>> https://www1.ietf.org/mailman/listinfo/bmwg
> 
> 
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www1.ietf.org/mailman/listinfo/bmwg
> 
> 

_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg