Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergence benchmarking
Scott Poretsky <sporetsky@avici.com> Fri, 14 February 2003 17:54 UTC
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28829 for <bmwg-archive@lists.ietf.org>; Fri, 14 Feb 2003 12:54:46 -0500 (EST)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1EHwEp11376; Fri, 14 Feb 2003 12:58:14 -0500
Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1EHvdp11345 for <bmwg@optimus.ietf.org>; Fri, 14 Feb 2003 12:57:39 -0500
Received: from mailhost.avici.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28797 for <bmwg@ietf.org>; Fri, 14 Feb 2003 12:53:22 -0500 (EST)
Received: from sporetsky-pc2.avici.com (sporetsky-pc2.avici.com [10.2.22.238]) by mailhost.avici.com (8.11.0/8.11.0) with ESMTP id h1EHv5d05551 for <bmwg@ietf.org>; Fri, 14 Feb 2003 12:57:05 -0500 (EST)
Message-Id: <5.0.2.1.2.20030214123729.08440ab0@mailhost.avici.com>
X-Sender: sporetsky@mailhost.avici.com
X-Mailer: QUALCOMM Windows Eudora Version 5.0.2
Date: Fri, 14 Feb 2003 12:57:53 -0500
To: bmwg@ietf.org
From: Scott Poretsky <sporetsky@avici.com>
Subject: Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergence benchmarking
In-Reply-To: <3E4D1254.8090007@juniper.net>
References: <5.0.2.1.2.20030210131816.01bfc250@mailhost.avici.com> <5.0.2.1.2.20030213103530.031b2090@mailhost.avici.com>
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary="=====================_336699458==_.ALT"
Sender: bmwg-admin@ietf.org
Errors-To: bmwg-admin@ietf.org
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
Folks, While I think that IGP Convergence testing is very worthwhile work for the BMWG to undertake, these drafts do not address IGP Convergence. These proposed drafts focus on LSA propagation benchmarking. Due to the extreme interest in industry for IGP convergence measurements, I was working on IGP Convergence terminology and benchmarking methodology drafts when these proposed drafts were submitted. I have provided the IGP Convergence drafts to Kevin and they should soon be posted. My comments to the proposed LSA Propagation benchmarking drafts are below. Scott ----------------------------------------------------------------------------- Terminology Internal Measurements and External Measurements are already defined as White Box and Black Box. Point-to-Point, Broadcast, SPF, Hello Interval, and Router Dead Interval are already defined in IETF RFCs Incremental-SPF is a vendor-specific feature. Convergence is stated as " A network is termed to be converged when all of the devices within the network have a loop free path to each possible destination. " The definition is then modified for a single DUT as follow "Since we are not testing network convergence, but performance for a particular device within a network, however, this definition needs to be narrowed somewhat to fit within a single device view. In this case, convergence will mean the point in time when the DUT has performed all actions needed to react to the change in topology represented by the test condition ". This must include updating the FIB and hardware as externally verified with data traffic. This draft contradicts its own definition by verifying convergence through measurement of only LSA propagation. In fact, updating the FIB is explicitly excluded from measurement. Methodology Section 5. Overview and Scope Control Plane measurements are not convergence measurements. Convergence is benchmarked by time to reroute traffic. Section 6. Test Conditions Bullet 2 It cannot be assumed that there is zero delay to execute the SPF or LSA processing. Reference [5] actually _recommends_ router vendors implement zero delay to achieve low convergence. The benchmarking methodology should allow convergence measurements to be made regardless of implementation. Bullet 3 Data traffic is actually required to make a deterministic measurement of route convergence. Unlisted: Number of IGP routes is a critical parameter for convergence time. Sections 7. through 9. The biggest problem with the methodology is that not all components for convergence time are measured. Focus is on LSA Processing, which is only 1 of 4 major components. Route change due to cost or next-hop change is not considered. This is the most common reason for route convergence. Well -defined Convergence methodology can be applied to ISIS or OSPF. The Link-State IGP does not matter. In fact the methodology should be able for use to benchmark one IGP against the other given the same DUT. All procedures meet one or more of the following flaws: White box measurements are part of the procedure. Time to install route (update FIB), update hardware, and reroute traffic are not considered. Test case is Implementation-specific. Test case has nothing to do with convergence - more OSPF performance test. Applicability The applicability document explains that convergence testing is needed using 1. White box benchmarks 2. DUT intrusive measurements 3. Knowledge of DUT implementation 4. Absence of traffic results in impractical benchmarks. Convergence time is benchmarked by traffic loss. 1 through 3 violate basic test methodology requirements. 4 makes invalid convergence tests. This work is less about convergence benchmarking and more about LSA propagation benchmarking. >>>>>At 08:42 PM 2/7/03 -0500, you wrote: >>>>> >>>>>>BMWG'ers: >>>>>> >>>>>>A WG Last Call period for the Internet-Drafts regarding >>>>>>OSPF convergence benchmarking terminology, methodology, and >>>>>>benchmark applicability: >>>>>> >>>>>> <draft-ietf-bmwg-ospfconv-term-02.txt>, >>>>>> <draft-ietf-bmwg-ospfconv-intraarea-03.txt>, >>>>>> <draft-ietf-bmwg-ospfconv-applicability-01.txt> >>>>>> >>>>>>will be open from 7 February 2003 until 28 February 2003. >>>>>> >>>>>>Please weigh in on whether or not you feel each individual draft >>>>>>should not be given to the Area Directors for consideration in >>>>>>progressing the draft to an Informational RFC. Send your comments >>>>>>to this list or kdubray@juniper.net. >>>>>> >>>>>> >>>>>>URLs for the Internet-Drafts are: >>>>>> >>>>>>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ospfconv-term-02.txt >>>>>>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ospfconv-intraarea-03.txt >>>>>> >>>>>>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ospfconv-applicability-01.txt >>>>>> >>>>>> >>>>>>-Kevin >>>>>> >>>>>> >>>>>>_______________________________________________ >>>>>>bmwg mailing list >>>>>>bmwg@ietf.org >>>>>>https://www1.ietf.org/mailman/listinfo/bmwg
- [bmwg] WG Last Call: OSPF convergence benchmarking Kevin Dubray
- Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergenc… Scott Poretsky
- Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergenc… Russ White
- RE: Fwd: Re: [bmwg] WG Last Call: OSPF convergenc… Manral, Vishwas
- Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergenc… Russ White
- Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergenc… Kevin Dubray
- [bmwg] WG Last Call: OSPF convergence benchmarking Kevin Dubray
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Al Morton
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Randy Bush
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Scott Poretsky
- Re: [bmwg] WG Last Call: OSPF convergence benchma… Russ White