Re: [bmwg] Comments for OSPF Convergence Benchmarking Terminology

> Internal Measurements - This is a redefinition of White Box measurement.
> Recommend it be removed.

The definition is not the major point of this defintion, but rather the
discussion about internal measurements, that they can impact the
performance of the box being measured, etc. I don't know if there is a
"generic" draft which covers these sorts of things, or if one is planned,
but it could go there just as well as here. In the meantime, it seems like
a useful discussion to have on paper.

> External Measurements - This is a redefinition of Black Box measurement.
> Recommend it be removed.  In the Discussion section for External
> Measurements it states:  "when a downstream device receives complete
> routing information from the DUT, it can be inferred that the DUT has
> transmitted all of the routing information available". How can tests be
> built so that it is not "inferred", but instead "known"?

You can't do more than infer, in most cases. You can state that it is
"known" that the DUT has finished sending information, but you don't know
if you are going to, for instance, drop the last ack, and force the DUT to
retransmit.

> Multi-Device Measurement - This is a definition describing a test with
> multiple DUTs.  This term serves no purpose.  In fact, it does not even
> appear in the companion methodology.  Recommend it be removed.

Actually, in our definition, a multi-device test requires measurement of
the test results on several devices, not necessarily several DUTs. For
instance, you may need to measure the timestamp on both the route generator
and the DUT, which means it's a multi-device mesasurement (not a
multi-device test, which would imply something different). I don't argue
that we didn't use the term in the methodology, but we certainly described
some multi-device measurements in some of the tests.

Again, the discussion is important here, in how some of these issues
surrounding multi-device measurements.

> Point-to-Point Link - Term was already defined by the Point-to-Point
> Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for
> the Transmission of Multi-Protocol Datagrams over Point-to-Point Links.
> Recommend it be removed.

First, the concept of a point-to-point link encompasses more link types
that the PPP working group covers. For instance, there is a draft in the
IS-IS and OSPF working groups which discusses point-to-point ethernet
links, and how these protocols should treat these links different than a
normal ethernet link. In this case, I'm not certain the duplicate defintion
is a bad thing, an the discussion about convergence times in OSPF vis-a-vis
point-to-point links needs a home.

> Broadcast Link - While this term is not defined, but frequently used, in
> IETF documents, it is a well-known term defined in IEEE 802 documents.
> Recommend it be removed.

Note the definition is in terms of OSPF, and the benchmark draft. Again, it
may be a commonly known term, but the concept of a DR being elected on the
link is something which is specific to OSPF.

> SPF Time - This is a white box parameter that is not externally
> observable.  RFC 2328 already defines SPF.  Recommend it be removed.

There are tests included which discuss how to measure SPF time from outside
the box, and inside the box. I'm fairly certain the methodology draft
includes white box measurements, so I'm not certain why white box
measurements shouldn't be included in the terminology draft?

> Hello Interval - This is a term already defined in RFC 2328, OSPFv2,
> Section 9, The Interface Data Structure and its use described in detail
> RFC 2328, OSPFv2, in Section 9.5, Sending Hello Packets and Section 10
> Neighbor Data Structure.  Recommend it be removed.

The discussion here is just as important as the definition, and applies to
all tests discussed in the methodology draft. Suggestions on where else we
would put this discussion?

> Router Dead Interval - This is a term already defined in RFC 2328,
> OSPFv2, Section 9, The Interface Data Structure and its use described in
> detail RFC 2328, OSPFv2, Section 10 Neighbor Data Structure.  Recommend
> it be removed.

See above, on hello interval.

> Incremental-SPF - This is a white box component of OSPF. Implementation
> of this feature is optional. Superior Convergence times can be obtained
> without Incremental-SPF (as two vendors have demonstrated).  Black box
> measurements cannot indicate if Incremental-SPF is internally implemented
> in the DUT.  RFC 2328 makes one reference to Incremental-SPF as an
> example algorithm to improve SPF run-time.  It does not require it nor
> indicate it to be the _most_ efficient algorithm.  If we were to
> standardize internal SPF execution algorithms then we should also include
> PRC, RTC, and others.  There would be no reason to choose just
> Incremental-SPF.  Recommend it be removed.

I don't see it used in either draft, so this is fine.

> Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This
> is a partial measurement of router components that contribute to
> Convergence time.  Time to update FIB is certainly a missing component.
> Also, network operators speak of "Convergence" as the time it takes for
> all nodes in the network to converge.  I propose that this term be
> renamed "Protocol Convergence" or "Control-Plane Convergence".

Either way is fine with me.... Of course, this definition draft only sets
the meaning of words for the accompanying methods and applicability drafts,
and not in general. Others have defined convergence differently, so.... I
suppose that we, as a working group, can go one of two directions on this:

-- We come up with a new name for convergence, prepending it with some
other word, to imply different things in different places. This could
probably lead to large and protracted arguments, and perhaps
overspecification of the meaning of words.

(for instance, we've already added "SR" to the front of convergence for a
similar objection, so now we'll have:

-- SR-CP-Convergence, Single Router Control Plane
-- N-CP-Convergence, Network Control Plane
-- SR-DP-Convergence, Single Router Data Plane
-- N-DP-Convergence, Network Data Plane

as a base set of variants, with, I assume, many many more to come).

-- We allow each draft to use the word "convergence" within it's bounding
limits (the dictionary definition), and just make certain that each draft
discusses its definition of convergence at some point, to clarify for the
readers of that draft. always remembering to have a flexible mind, and seek
out the meaning of convergence before reading any given BMWG draft.

I'll defer to the working group for that discussion and decsion, though I
prefer the latter.

> 2. Measuring Traffic Convergence Section
> Three measurements are listed.  Here are the concerns:
> A. LSA Propagation
> B. SPF Execution - This is a White Box Measurement

A test is included in the methods draft, so I'm not certain why this
shouldn't be here?

> C. RIB Update - How can this be measured in isolation from 1 and 2
> without it being a white-box test?

We don't include a test for this, so whether it is white box or black box
makes no difference? It's just noted as a part of overal convergence times,
to be complete in the discussion of the elements of convergence time.

> 3. Type of Network Events Section
> Next-Hop change (e.g. due to metric change) is a unique event because no
> packet loss is expected, so this should also be listed.  The other events
> listed - Link Up/Down, Session Initialization, and Adjacency Down
> produce packet loss.

We didn't discuss packet loss in any section, since we are concerned with
contorl plane convergence, rather than data plane convergence (?). We can
add next-hop change, although I'm not certain we base any tests on it, and
making this list exhasutive would, well, unless we want to do that as a
generic document, we probably don't want to go there. :-)

> 4. LSA and Destination Mix
> No new terminology.  Methodology related information.  Recommend move to
> methodology document.

This could also be moved to the applicability document, but we tried to
move these sorts of discussions outside either one of those docs, since
they didn't seem to match the flow of the documents themselves. I don't
know, actually, where we would put them in the methdology draft--it would
have to either go in the front, in a general considerations section of some
type, or it would have to go in the individual tests, repeated as needed,
or at the end, where it likely wouldn't get read.

I'll defer to the working group on this one--I see no compelling reason to
move them or to leave them.

> 5. Tree Shape and SPF algorithm
> These are White Box items.  Neither is externally observable.  Tree shape
> is already described in RFC 2328.  I recommend that this section be
> removed.  It could be added as an appendix to the methodology document, but
> that is not worth it since it is already covered in RFC 2328.

First, I'm not certain where we are defining these drafts as black box
only? We refer to a paper on black box measurements, but I don't think
this methodology draft is confined to black box measurements. There are
specific warnings in various discussions about using white box
measurements, and the problems you can have with them.

Second, the concept of tree depth and tree breadth might be considered in
RFC2328, but I'm not certain it's not worth discussing it again, from the
point of view of the impact the shape of the topology you feed to the
routers can have on the test results.

As for where it should be, again, I see no reason to move it or not move
it.... I'll defer to the chair and the rest of the working group on this
one.

:-)

Russ

-- 
riw@cisco.com CCIE <>< Grace Alone
_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg