[bmwg] Initial Reply to Publication Request on igp-dataplane drafts

[Al Morton <[e-mail]>]

BMWG,

Below are Ron Bonica's comments on the terminology draft
so far (through section 3.12). We have some work to do to
clarify many of the definitions. It's good to have these
comments from a fresh perspective -- significant improvements
are now possible!

Al
bmwg co-chair

The following are comments on
draft-ietf-bmwg-igp-dataplane-conv-term-12. On the whole, this is a very
good document. My comments are a bit lawyerish ;-)

Please look for the prefix RB>>

                           Ron





    Network Working Group
    INTERNET-DRAFT
    Expires in: August 2007
    Intended Status: Informational
                                                    Scott Poretsky
                                                    Reef Point Systems

                                                    Brent Imhoff
                                                    Juniper Networks

                                                    February 2007

                         Terminology for Benchmarking
                       IGP Data Plane Route Convergence

                 <draft-ietf-bmwg-igp-dataplane-conv-term-12.txt>

Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.

RB> Formatting issue

Status of this Memo

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF), its areas, and its working groups.  Note that
    other groups may also distribute working documents as
    Internet-Drafts.

    Internet-Drafts are draft documents valid for a maximum of six months
    and may be updated, replaced, or obsoleted by other documents at any
    time.  It is inappropriate to use Internet-Drafts as reference
    material or to cite them other than as "work in progress."

    The list of current Internet-Drafts can be accessed at
    http://www.ietf.org/ietf/1id-abstracts.txt.

    The list of Internet-Draft Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html.

Copyright Notice
    Copyright (C) The IETF Trust (2007).

ABSTRACT
    This document describes the terminology for benchmarking Interior
    Gateway Protocol (IGP) Route Convergence.   The terminology is to
    be used for benchmarking IGP convergence time through externally
    observable (black box) data plane measurements.  The terminology
    can be applied to any link-state IGP, such as ISIS and OSPF.

Poretsky, Imhoff                                                [Page 1]
INTERNET-DRAFT           Benchmarking Terminology for      February 2007
                        IGP Data Plane Route Convergence

Table of Contents
      1. Introduction .................................................2
      2. Existing definitions .........................................3
      3. Term definitions..............................................3
         3.1 Convergence Event.........................................3
         3.2 Route Convergence.........................................4
         3.3 Network Convergence.......................................4
         3.4 Full Convergence..........................................5
         3.5 Convergence Packet Loss...................................5
         3.6 Convergence Event Instant.................................6
         3.7 Convergence Recovery Instant..............................6

         3.8 Rate-Derived Convergence Time.............................7
         3.9 Convergence Event Transition..............................7
         3.10 Convergence Recovery Transition..........................8
         3.11 Loss-Derived Convergence Time............................8
         3.12 Sustained Forwarding Convergence Time....................9
         3.13 Restoration Convergence Time.............................9
         3.14 Packet Sampling Interval.................................10
         3.15 Local Interface..........................................11
         3.16 Neighbor Interface.......................................11
         3.17 Remote Interface.........................................11
         3.18 Preferred Egress Interface...............................12
         3.19 Next-Best Egress Interface...............................12
         3.20 Stale Forwarding.........................................13
         3.21 Nested Convergence Events................................13
      4. IANA Considerations...........................................13
      5. Security Considerations.......................................14
      6. Acknowledgements..............................................14
      7. Normative References..........................................14
      8. Author's Address..............................................15

1. Introduction
    This draft describes the terminology for benchmarking Interior
    Gateway Protocol (IGP) Route Convergence.  The motivation and
    applicability for this benchmarking is provided in [Po07a].  The
    methodology to be used for this benchmarking is described in [Po07m].
    The methodology and terminology to be used for benchmarking Route
    Convergence can be applied to any link-state IGP such as ISIS [Ca90]
    and OSPF [Mo98].  The data plane is measured to obtain black-box
    (externally observable) convergence benchmarking metrics.  The
    purpose of this document is to introduce new terms required to
    complete execution of the IGP Route Convergence Methodology [Po07m].
    These terms apply to IPv4 and IPv6 traffic and IGPs.

    An example of Route Convergence as observed and measured from the
    data plane is shown in Figure 1.  The graph in Figure 1 shows
    Forwarding Rate versus Time.  Time 0 on the X-axis is on the far
    right of the graph.  The Offered Load to the ingress interface of
    the DUT SHOULD equal the measured maximum Throughput [Ba99][Ma98]
    of the DUT and the Forwarding Rate [Ma98] is measured at the egress
    interfaces of the DUT.   The components of the graph and the metrics
    are defined in the Term Definitions section.

Poretsky, Imhoff                                                [Page 2]
INTERNET-DRAFT           Benchmarking Terminology for      February 2007
                        IGP Data Plane Route Convergence


                         Convergence    Convergence
                         Recovery       Event
                         Instant        Instant  Time = 0sec
    Forwarding Rate =       ^              ^       ^     Offered Load =
      Offered Load --> ------\    Packet   /-------- <---Max Throughput
                              \    Loss   /<----Convergence
            Convergence------->\         /      Event Transition
         Recovery Transition    \       /
                                 \_____/<------Maximum Packet Loss
         X-axis = Time
         Y-axis = Forwarding Rate

                         Figure 1. Convergence Graph

2.  Existing definitions
    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
    document are to be interpreted as described in BCP 14, RFC 2119
    [Br97].  RFC 2119 defines the use of these key words to help make the
    intent of standards track documents as clear as possible.  While this
    document uses these keywords, this document is not a standards track
    document.  The term Throughput is defined in [Ba91] and [Ba99].

3. Term Definitions
    3.1 Convergence Event

         Definition:
         The occurrence of a planned or unplanned action in the network
         that results in a change in the egress interface of the Device
         Under Test (DUT) for
         routed packets.

         Discussion:
         Convergence Events include link loss, routing protocol session
         loss, router failure, configuration change, and better next-hop
         learned via a routing protocol.

         Measurement Units:
         N/A

         Issues:
         None

         See Also:
         Convergence Packet Loss
         Convergence Event Instant





Poretsky, Imhoff                                                [Page 3]
INTERNET-DRAFT           Benchmarking Terminology for      February 2007
                        IGP Data Plane Route Convergence

    3.2 Route Convergence

         Definition:
         Recovery from a Convergence Event indicated by the DUT
         Throughput equal to the offered load.

         Discussion:
         Route Convergence is the action of all components of the router
         being updated with the most recent route change(s) including the
         Routing Information Base (RIB) and Forwarding Information Base
         (FIB), along with software and hardware tables.  Route
         Convergence can be observed externally by the rerouting of data
         Traffic to a new egress interface.

         Measurement Units:
         N/A

         Issues:
         None

         See Also:
         Network Convergence
         Full Convergence
         Convergence Event

RB> I'm not sure that I understand the difference between route
convergence and full convergence. Does route convergence apply to a
single route while full convergence applies to the entire FIB? You might
want to clarify that.

RB> Also, you might want to point out that route convergence isn't
always the same as sustained route convergence.

RB> Also, your definition of route convergence requires only that the
DUT is sending packets *someplace*. There is no requirement that the DUT
send the packet to *the right place*. Consider the following problem:

RB> Let's say that the DUT has three interfaces, A, B, C. Before the
convergence event, traffic enters the DUT through A and leaves through
B. But B goes down, causing a transient condition in which the DUT
forwards packets right back out through A. During this transient period,
packets loop until the TTL expires. (This doesn't help anybody much!) At
the end of the transient period, routing converges and the DUT forwards
packets out through Interface C, whence they have a fighting chance of
delivery. So, when did route convergence occur, at the beginning or the
transient period or at the end? According to your definition, it
happened at the beginning.


    3.3 Network Convergence

         Definition:
         The completion of updating of all routing tables, including the
         FIB, in all routers throughout the network.

         Discussion:
         Network Convergence is bounded by the sum of Route Convergence
         for all routers in the network.  Network Convergence can be
         determined by recovery of the Throughput to equal the offered
         load, with no Stale Forwarding, and no blenders [Ca01][Ci03].

RB> Above, you defined Route Convergence as "Recovery from a Convergence
Event indicated by the DUT Throughput equal to the offered load". This
is an event, not a quantity. So, you can't talk about the "the sum of
Route Convergence".

         Measurement Units:
         N/A

         Issues:
         None

         See Also:
         Route Convergence
         Stale Forwarding




Poretsky, Imhoff                                                [Page 4]
INTERNET-DRAFT           Benchmarking Terminology for      February 2007
                        IGP Data Plane Route Convergence

    3.4 Full Convergence
         Definition:
         Route Convergence for an entire FIB.

         Discussion:
         When benchmarking convergence, it is useful to measure
         the time to converge an entire FIB.  For example,
         a Convergence Event can be produced for an OSPF table of
         5000 routes so that the time to converge routes 1 through
         5000 is measured.  Full Convergence is externally observable
         from the data plane when the Throughput of the data
         plane traffic on the Next-Best Egress Interface equals the
         offered load.

         Measurement Units:
         N/A

         Issues: None

         See Also:
         Network Convergence
         Route Convergence
         Convergence Event

    3.5 Convergence Packet Loss

         Definition:
         The amount of packet loss produced by a Convergence Event
         until Route Convergence occurs.

RB> Units of measure?

         Discussion:
         Packet loss can be observed as a reduction of forwarded traffic
         from the maximum Throughput.  Convergence Packet Loss
         includes packets that were lost and packets that were delayed
         due to buffering.

RB> How will you count the packets that were delayed? Timestamping each
packet? How much delay will you tolerate before you say that a packet is
lost?

RB> Isn't this metric influence by the rate at which you send packets
into the DUT. If the pause between packets is long enough, you may not
observe any loss between the convergence event and sustained route
convergence.


   The maximum Convergence Packet Loss observed
         in a Packet Sampling Interval may or may not reach 100% during
         Route Convergence (see Figure 1).

         Measurement Units:
         number of packets

RB> In 3.11 you say that the unit of measure is packets per second.
Which is it?

         Issues: None

         See Also:
         Route Convergence
         Convergence Event
         Rate-Derived Convergence Time
         Loss-Derived Convergence Time
         Packet Sampling Interval

Poretsky, Imhoff                                                [Page 5]
INTERNET-DRAFT             Benchmarking Terminology for    February 2007
                          IGP Data Plane Route Convergence

    3.6 Convergence Event Instant

         Definition:
         The time instant that a Convergence Event becomes observable in
         the data plane.

         Discussion:
         Convergence Event Instant is observable from the data
         plane as the precise time that the device under test begins
         to exhibit packet loss.

         Measurement Units:
         hh:mm:ss:nnn, where 'nnn' is milliseconds

         Issues:
         None

         See Also:
         Convergence Event
         Convergence Packet Loss
         Convergence Recovery Instant

    3.7 Convergence Recovery Instant

         Definition:
         The time instant that Full Convergence is measured
         and then maintained for an interval of duration equal to
         the Sustained Forwarding Convergence Time

         Discussion:
         Convergence Recovery Instant is measurable from the data
         plane as the precise time that the device under test
         achieves Full Convergence.

         Measurement Units:
         hh:mm:ss:uuu

         Issues:
         None

         See Also:
         Sustained Forwarding Convergence Time
         Convergence Packet Loss
         Convergence Event Instant








Poretsky, Imhoff                                                [Page 6]
INTERNET-DRAFT             Benchmarking Terminology for    February 2007
                          IGP Data Plane Route Convergence

    3.8 Rate-Derived Convergence Time
         Definition:
         The amount of time for Convergence Packet Loss to persist upon
         occurrence of a Convergence Event until occurrence of Route
         Convergence.

         Rate-Derived Convergence Time can be measured as the time
         difference from the Convergence Event Instant to the
         Convergence Recovery Instant, as shown with Equation 1.

         Equation 1 -
           Rate-Derived Convergence Time =
               Convergence Recovery Instant - Convergence Event Instant.

         Discussion:
         Rate-Derived Convergence Time should be measured at the maximum
         Throughput.  Failure to achieve Full Convergence results in
         a Rate-Derived Convergence Time benchmark of infinity.

         Measurement Units:
         seconds/milliseconds

         Issues:
         None

         See Also:
         Convergence Packet Loss
         Convergence Recovery Instant
         Convergence Event Instant
         Full Convergence

    3.9 Convergence Event Transition
         Definition:
         The characteristic of a router in which Throughput
         gradually reduces to zero after a Convergence Event.

RB> Is this a period of time or the characteristic of a router? If it is
a characteristic, why does it have units of measure.

RB> If it is a period of time, doesn't the packet sampling interval and
data rate of the test stream influence it?

         Discussion:
         The Convergence Event Transition is best observed for
         Full Convergence.  The Convergence Event Transition may
         not be linear.

         Measurement Units:
         seconds/milliseconds

         Issues:
         None

         See Also:
         Convergence Event
         Rate-Derived Convergence Time
         Convergence Packet Loss
         Convergence Recovery Transition

Poretsky, Imhoff                                                [Page 7]
INTERNET-DRAFT             Benchmarking Terminology for    February 2007
                          IGP Data Plane Route Convergence

    3.10 Convergence Recovery Transition

         Definition:
         The characteristic of a router in which Throughput
         gradually increases to equal the offered load.

RB> Same comments as above.

         Discussion:
         The Convergence Recovery Transition is best observed for
         Full Convergence.  The Convergence Event Transition may
         not be linear.

         Measurement Units:
         seconds/milliseconds

         Issues: None

         See Also:
         Full Convergence
         Rate-Derived Convergence Time
         Convergence Packet Loss
         Convergence Event Transition

    3.11 Loss-Derived Convergence Time

         Definition:
         The amount of time it takes for Route Convergence to
         to be achieved as calculated from the Convergence Packet
         Loss.  Loss-Derived Convergence Time can be calculated
         from Convergence Packet Loss that occurs due to a
         Convergence Event and Route Convergence as shown with
         Equation 2.

         Equation 2 -
           Loss-Derived Convergence Time =
                 Convergence Packets Loss / Offered Load
                 NOTE: Units for this measurement are
                 packets / packets/second = seconds

         Discussion:
         Loss-Derived Convergence Time gives a better than
         actual result when converging many routes simultaneously.
         Rate-Derived Convergence Time takes the Convergence Recovery
         Transition into account, but Loss-Derived Convergence Time
         ignores the Route Convergence Recovery Transition because
         it is obtained from the measured Convergence Packet Loss.

         Ideally, the Convergence Event Transition and Convergence
         Recovery Transition are instantaneous so that the
         Rate-Derived Convergence Time = Loss-Derived Convergence Time.
         However, router implementations are less than ideal.
         For these reasons the preferred reporting benchmark for IGP
         Route Convergence is the Rate-Derived Convergence Time.

Poretsky, Imhoff                                                [Page 8]
INTERNET-DRAFT             Benchmarking Terminology for    February 2007
                          IGP Data Plane Route Convergence

         Guidelines for reporting Loss-Derived Convergence Time are
         provided in [Po07m].

         Measurement Units:
         seconds/milliseconds

         Issues:
         None

         See Also:
         Route Convergence
         Convergence Packet Loss
         Rate-Derived Convergence Time
         Convergence Event Transition
         Convergence Recovery Transition

    3.12 Sustained Forwarding Convergence Time

         Definition:
         The amount of time for which Full Convergence is maintained
         without additional packet loss.

         Discussion:
         The purpose of the Sustained Forwarding Convergence Time is to
         produce Convergence benchmarks protected against fluctuation
         in Throughput after Full Convergence is observed.  The
         Sustained Forwarding Convergence Time to be used is calculated
         as shown in Equation 3.

         Equation 3 -
         Sustained Forwarding Convergence Time =
             5*(Convergence Packet Loss/Offered Load)
                units are packets/pps = sec

RB> Not sure that I understand where the constant 5 came from.

         for which at least one packet per route in the FIB for all
         routes in the FIB MUST be offered to the DUT per second.

         Measurement Units:
         seconds or milliseconds

         Issues: None

         See Also:
         Full Convergence
         Convergence Recovery Instant


_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg