RE: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing
"Shankar Rao" <srao@qmail.qwest.net> Fri, 09 May 2003 20:33 UTC
Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00933 for <bmwg-archive@lists.ietf.org>; Fri, 9 May 2003 16:33:41 -0400 (EDT)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h49KeN810921; Fri, 9 May 2003 16:40:23 -0400
Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h49KCk808798 for <bmwg@optimus.ietf.org>; Fri, 9 May 2003 16:12:46 -0400
Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00283 for <bmwg@ietf.org>; Fri, 9 May 2003 16:02:10 -0400 (EDT)
Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19EE6G-00075i-00 for bmwg@ietf.org; Fri, 09 May 2003 16:04:12 -0400
Received: from uswgne22.uswest.com ([204.26.87.76]) by ietf-mx with esmtp (Exim 4.12) id 19EE6F-00075c-00 for bmwg@ietf.org; Fri, 09 May 2003 16:04:11 -0400
Received: from egate-co5.uswc.uswest.com (egate-co5.uswc.uswest.com [151.119.87.232]) by uswgne22.uswest.com (8.xx.x/8.xx.x) with ESMTP id h49K4ZBF027337; Fri, 9 May 2003 15:04:35 -0500 (CDT)
Received: from denntex012.qwest.net (localhost [127.0.0.1]) by egate-co5.uswc.uswest.com (8.12.8/8.12.8) with ESMTP id h49K4Zi5027341; Fri, 9 May 2003 14:04:35 -0600 (MDT)
Received: from srao (srao.zone0.qintra.com [10.0.84.152]) by denntex012.qwest.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id KGAXD54Y; Fri, 9 May 2003 14:04:34 -0600
Reply-To: srao@qwest.com
From: Shankar Rao <srao@qmail.qwest.net>
To: 'Kevin Dubray' <kdubray@juniper.net>, bmwg@ietf.org
Cc: "'Perser, Jerry'" <jerry.perser@spirentcom.com>
Subject: RE: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing
Date: Fri, 09 May 2003 14:04:35 -0600
Organization: Qwest
Message-ID: <000801c31666$36a51690$9854000a@AD.QINTRA.COM>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.2627
Importance: Normal
In-Reply-To: <3EBAC4F0.5010206@juniper.net>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Content-Transfer-Encoding: 7bit
Sender: bmwg-admin@ietf.org
Errors-To: bmwg-admin@ietf.org
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7bit
Kevin, This work, in my opinion is clearly a benchmarking test, like any other test that may measure benchmarks such as through throughput, latency, frame loss rate, system recovery, reset etc. This effort will measure similar benchmarks in the context of each of the configuration modules proposed (control, data, management, security). The only differnce will be that such benchmarking parameters will be gathered under a predefined steady state. For example, lets consider the routing protocol configuration set under control plane module. Our measurement units for the ISIS routing protocol include (Section 3.2.1.1) - ISIS Enabled/Disabled ISIS-TE Enabled/Disabled Number of ISIS Adjacencies Adjacencies Number of ISIS Routes Routes Number of Nodes per Area Nodes Each of these measurement units can contribute to a becnhmark test. Lets say we want to benchmark the max. number of ISIS routes. One way of doing this is to run only ISIS on the DUT (and nothing else) and determine the maximum number of ISIS routes supported with 0% frame loss rate. Another way might be to do the same test, with a predefined steady state consisting of all the other protocols that DUT may be required to support (e.g. BGP, RSVP, etc) and gathering the same test data - how many ISIS routes did the DUT support under these modified steady state conditions at 0% frame loss rate? Regarding the concern about variability in test results, that by itself is a benchmark in my opinion. If the DUT does exhibit widely variable behavior, it is very concerning from an operational standpoint. Predictability and a uniform response to stimulus is as important as bare functionality. In addition, even though it may be the case that increasing the number of workload factors makes it harder to get a handle on what's going on, the reality is that the real world works that way - most carriers are running multiple protocols on their core devices, today. It is concerning for me as a service provider if I hear that we, as a working group cannot define how we can get a handle of what is happening on such core routers even in a controlled environment. Shankar. -----Original Message----- From: bmwg-admin@ietf.org [mailto:bmwg-admin@ietf.org] On Behalf Of Kevin Dubray Sent: Thursday, May 08, 2003 2:58 PM To: bmwg@ietf.org Cc: Perser, Jerry Subject: Re: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing Perser, Jerry wrote: > Kevin, > > I see Scott going down one of two paths: either this proposal is an > interesting software verification test, or a mixed benchmark. > > I believe the title is worded such that it looks like a software > verification test. If the path is an interesting test, it does fit in > the BMWG. I think the ADs might have something to say about the last premise. "Interesting" isn't the sole entry condition for this WG. > On the other hand, if this is a procedure for mixing previous > benchmarking RFCs, there is a desire to use it. We have an RFC to > benchmark walking. We have a benchmark for chewing gum. But what > happens when you walk and chew gum? We should focus on the metrics, > not if we swallow the gum. It's my experience, that as you increase the number of workload factors applied to a system under test, the harder it is to keep a handle on what's going on. Moreover, exacting replication of the workload is a non-trivial exercise with a large number of workload factors. Often, responses have a wide degree fo variability - run the test once, get one response; run it again, get another response. Sometimes, this is an artifact of SUT behavior, but sometimes the test equipment behaves less than uniformly, especially if we start mixing too many metaphors. :-) Personally, I think this is a great test. I also believe that the proposed effort presents too many knobs to successfully produce a dependable, uniform yardstick from which to gather comparative [core] router reliability measurements. > It would be valueable work if we expand mixing to more than unicast > and multicast frames. The mcastm-12 I-D standardizes how to benchmark > two classes of frames. How about we expand this concept to other > benchmarks on a core router? I'm sure there are many other factors that could be added to make the operational profile more interesting; however, it also exacerbates the difficulties highlighted above. In lieu of this benchmarking forum, I believe the proposed topic is better specified in a provider's validation test plan, where applicable context can be specifically tailored to a provider's individual operational requirements as well as its test equipments' capability. -Kevin _______________________________________________ bmwg mailing list bmwg@ietf.org https://www1.ietf.org/mailman/listinfo/bmwg _______________________________________________ bmwg mailing list bmwg@ietf.org https://www1.ietf.org/mailman/listinfo/bmwg
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… Michele Bustos
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… Tony DeLaRosa
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… ananda_sengupta
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… john_nakulski
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… Jim McQuaid
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… Perser, Jerry
- Re: [bmwg] New WG Deliverable Proposal - Core Rou… Kevin Dubray
- RE: [bmwg] New WG Deliverable Proposal - Core Rou… Shankar Rao