RE: [bmwg] Comments for OSPF Convergence Benchmarking Terminology

["Manral, Vishwas" <[e-mail]>]

Hi Scott,

Sorry to budge in, but I would like to comment on this.

We initially had only a white-box test for SPF capculations, which used show
commands(which are readily available in most routers). However as in the
working group we use black-box measurements, we added another test to
measure the SPF time thru black-box methods(which is the same test as in the
paper I have referred to in previous mails).

In the 53rd meeting of IETF in MNP, it was decided(if I understood it right)
that we keep the test more from the perspective of actually checking
white-box measurements against the actual black-box measurements. We have
used both these methods in our measurements, and have left both the methods
in the draft.

Thanks,
Vishwas 

-----Original Message-----
From: Scott Poretsky [mailto:[e-mail]]
Sent: Tuesday, March 11, 2003 9:08 PM
To: Russ White
Cc: [e-mail]
Subject: Re: [bmwg] Comments for OSPF Convergence Benchmarking
Terminology


Kevin,

I thought BMWG drafts are not permitted to have White Box test 
coverage.  Is this correct?  I also thought that existing terminology 
should either be excluded or referenced in the Existing Terms section.  Is 
this correct?

Scott

At 06:24 PM 3/10/03 -0500, Russ White wrote:

> > Internal Measurements - This is a redefinition of White Box measurement.
> > Recommend it be removed.
>
>The definition is not the major point of this defintion, but rather the
>discussion about internal measurements, that they can impact the
>performance of the box being measured, etc. I don't know if there is a
>"generic" draft which covers these sorts of things, or if one is planned,
>but it could go there just as well as here. In the meantime, it seems like
>a useful discussion to have on paper.
>
> > External Measurements - This is a redefinition of Black Box measurement.
> > Recommend it be removed.  In the Discussion section for External
> > Measurements it states:  "when a downstream device receives complete
> > routing information from the DUT, it can be inferred that the DUT has
> > transmitted all of the routing information available". How can tests be
> > built so that it is not "inferred", but instead "known"?
>
>You can't do more than infer, in most cases. You can state that it is
>"known" that the DUT has finished sending information, but you don't know
>if you are going to, for instance, drop the last ack, and force the DUT to
>retransmit.
>
> > Multi-Device Measurement - This is a definition describing a test with
> > multiple DUTs.  This term serves no purpose.  In fact, it does not even
> > appear in the companion methodology.  Recommend it be removed.
>
>Actually, in our definition, a multi-device test requires measurement of
>the test results on several devices, not necessarily several DUTs. For
>instance, you may need to measure the timestamp on both the route generator
>and the DUT, which means it's a multi-device mesasurement (not a
>multi-device test, which would imply something different). I don't argue
>that we didn't use the term in the methodology, but we certainly described
>some multi-device measurements in some of the tests.
>
>Again, the discussion is important here, in how some of these issues
>surrounding multi-device measurements.
>
> > Point-to-Point Link - Term was already defined by the Point-to-Point
> > Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for
> > the Transmission of Multi-Protocol Datagrams over Point-to-Point Links.
> > Recommend it be removed.
>
>First, the concept of a point-to-point link encompasses more link types
>that the PPP working group covers. For instance, there is a draft in the
>IS-IS and OSPF working groups which discusses point-to-point ethernet
>links, and how these protocols should treat these links different than a
>normal ethernet link. In this case, I'm not certain the duplicate defintion
>is a bad thing, an the discussion about convergence times in OSPF vis-a-vis
>point-to-point links needs a home.
>
> > Broadcast Link - While this term is not defined, but frequently used, in
> > IETF documents, it is a well-known term defined in IEEE 802 documents.
> > Recommend it be removed.
>
>Note the definition is in terms of OSPF, and the benchmark draft. Again, it
>may be a commonly known term, but the concept of a DR being elected on the
>link is something which is specific to OSPF.
>
> > SPF Time - This is a white box parameter that is not externally
> > observable.  RFC 2328 already defines SPF.  Recommend it be removed.
>
>There are tests included which discuss how to measure SPF time from outside
>the box, and inside the box. I'm fairly certain the methodology draft
>includes white box measurements, so I'm not certain why white box
>measurements shouldn't be included in the terminology draft?
>
> > Hello Interval - This is a term already defined in RFC 2328, OSPFv2,
> > Section 9, The Interface Data Structure and its use described in detail
> > RFC 2328, OSPFv2, in Section 9.5, Sending Hello Packets and Section 10
> > Neighbor Data Structure.  Recommend it be removed.
>
>The discussion here is just as important as the definition, and applies to
>all tests discussed in the methodology draft. Suggestions on where else we
>would put this discussion?
>
> > Router Dead Interval - This is a term already defined in RFC 2328,
> > OSPFv2, Section 9, The Interface Data Structure and its use described in
> > detail RFC 2328, OSPFv2, Section 10 Neighbor Data Structure.  Recommend
> > it be removed.
>
>See above, on hello interval.
>
> > Incremental-SPF - This is a white box component of OSPF. Implementation
> > of this feature is optional. Superior Convergence times can be obtained
> > without Incremental-SPF (as two vendors have demonstrated).  Black box
> > measurements cannot indicate if Incremental-SPF is internally
implemented
> > in the DUT.  RFC 2328 makes one reference to Incremental-SPF as an
> > example algorithm to improve SPF run-time.  It does not require it nor
> > indicate it to be the _most_ efficient algorithm.  If we were to
> > standardize internal SPF execution algorithms then we should also
include
> > PRC, RTC, and others.  There would be no reason to choose just
> > Incremental-SPF.  Recommend it be removed.
>
>I don't see it used in either draft, so this is fine.
>
> > Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This
> > is a partial measurement of router components that contribute to
> > Convergence time.  Time to update FIB is certainly a missing component.
> > Also, network operators speak of "Convergence" as the time it takes for
> > all nodes in the network to converge.  I propose that this term be
> > renamed "Protocol Convergence" or "Control-Plane Convergence".
>
>Either way is fine with me.... Of course, this definition draft only sets
>the meaning of words for the accompanying methods and applicability drafts,
>and not in general. Others have defined convergence differently, so.... I
>suppose that we, as a working group, can go one of two directions on this:
>
>-- We come up with a new name for convergence, prepending it with some
>other word, to imply different things in different places. This could
>probably lead to large and protracted arguments, and perhaps
>overspecification of the meaning of words.
>
>(for instance, we've already added "SR" to the front of convergence for a
>similar objection, so now we'll have:
>
>-- SR-CP-Convergence, Single Router Control Plane
>-- N-CP-Convergence, Network Control Plane
>-- SR-DP-Convergence, Single Router Data Plane
>-- N-DP-Convergence, Network Data Plane
>
>as a base set of variants, with, I assume, many many more to come).
>
>-- We allow each draft to use the word "convergence" within it's bounding
>limits (the dictionary definition), and just make certain that each draft
>discusses its definition of convergence at some point, to clarify for the
>readers of that draft. always remembering to have a flexible mind, and seek
>out the meaning of convergence before reading any given BMWG draft.
>
>I'll defer to the working group for that discussion and decsion, though I
>prefer the latter.
>
> > 2. Measuring Traffic Convergence Section
> > Three measurements are listed.  Here are the concerns:
> > A. LSA Propagation
> > B. SPF Execution - This is a White Box Measurement
>
>A test is included in the methods draft, so I'm not certain why this
>shouldn't be here?
>
> > C. RIB Update - How can this be measured in isolation from 1 and 2
> > without it being a white-box test?
>
>We don't include a test for this, so whether it is white box or black box
>makes no difference? It's just noted as a part of overal convergence times,
>to be complete in the discussion of the elements of convergence time.
>
> > 3. Type of Network Events Section
> > Next-Hop change (e.g. due to metric change) is a unique event because no
> > packet loss is expected, so this should also be listed.  The other
events
> > listed - Link Up/Down, Session Initialization, and Adjacency Down
> > produce packet loss.
>
>We didn't discuss packet loss in any section, since we are concerned with
>contorl plane convergence, rather than data plane convergence (?). We can
>add next-hop change, although I'm not certain we base any tests on it, and
>making this list exhasutive would, well, unless we want to do that as a
>generic document, we probably don't want to go there. :-)
>
> > 4. LSA and Destination Mix
> > No new terminology.  Methodology related information.  Recommend move to
> > methodology document.
>
>This could also be moved to the applicability document, but we tried to
>move these sorts of discussions outside either one of those docs, since
>they didn't seem to match the flow of the documents themselves. I don't
>know, actually, where we would put them in the methdology draft--it would
>have to either go in the front, in a general considerations section of some
>type, or it would have to go in the individual tests, repeated as needed,
>or at the end, where it likely wouldn't get read.
>
>I'll defer to the working group on this one--I see no compelling reason to
>move them or to leave them.
>
> > 5. Tree Shape and SPF algorithm
> > These are White Box items.  Neither is externally observable.  Tree
shape
> > is already described in RFC 2328.  I recommend that this section be
> > removed.  It could be added as an appendix to the methodology document,
but
> > that is not worth it since it is already covered in RFC 2328.
>
>First, I'm not certain where we are defining these drafts as black box
>only? We refer to a paper on black box measurements, but I don't think
>this methodology draft is confined to black box measurements. There are
>specific warnings in various discussions about using white box
>measurements, and the problems you can have with them.
>
>Second, the concept of tree depth and tree breadth might be considered in
>RFC2328, but I'm not certain it's not worth discussing it again, from the
>point of view of the impact the shape of the topology you feed to the
>routers can have on the test results.
>
>As for where it should be, again, I see no reason to move it or not move
>it.... I'll defer to the chair and the rest of the working group on this
>one.
>
>:-)
>
>Russ
>
_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg