RE: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing

"Shankar Rao" <srao@qmail.qwest.net> Fri, 09 May 2003 20:33 UTC

Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00933 for <bmwg-archive@lists.ietf.org>; Fri, 9 May 2003 16:33:41 -0400 (EDT)
Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h49KeN810921; Fri, 9 May 2003 16:40:23 -0400
Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h49KCk808798 for <bmwg@optimus.ietf.org>; Fri, 9 May 2003 16:12:46 -0400
Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00283 for <bmwg@ietf.org>; Fri, 9 May 2003 16:02:10 -0400 (EDT)
Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19EE6G-00075i-00 for bmwg@ietf.org; Fri, 09 May 2003 16:04:12 -0400
Received: from uswgne22.uswest.com ([204.26.87.76]) by ietf-mx with esmtp (Exim 4.12) id 19EE6F-00075c-00 for bmwg@ietf.org; Fri, 09 May 2003 16:04:11 -0400
Received: from egate-co5.uswc.uswest.com (egate-co5.uswc.uswest.com [151.119.87.232]) by uswgne22.uswest.com (8.xx.x/8.xx.x) with ESMTP id h49K4ZBF027337; Fri, 9 May 2003 15:04:35 -0500 (CDT)
Received: from denntex012.qwest.net (localhost [127.0.0.1]) by egate-co5.uswc.uswest.com (8.12.8/8.12.8) with ESMTP id h49K4Zi5027341; Fri, 9 May 2003 14:04:35 -0600 (MDT)
Received: from srao (srao.zone0.qintra.com [10.0.84.152]) by denntex012.qwest.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id KGAXD54Y; Fri, 9 May 2003 14:04:34 -0600
Reply-To: srao@qwest.com
From: Shankar Rao <srao@qmail.qwest.net>
To: 'Kevin Dubray' <kdubray@juniper.net>, bmwg@ietf.org
Cc: "'Perser, Jerry'" <jerry.perser@spirentcom.com>
Subject: RE: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing
Date: Fri, 09 May 2003 14:04:35 -0600
Organization: Qwest
Message-ID: <000801c31666$36a51690$9854000a@AD.QINTRA.COM>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.2627
Importance: Normal
In-Reply-To: <3EBAC4F0.5010206@juniper.net>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Content-Transfer-Encoding: 7bit
Sender: bmwg-admin@ietf.org
Errors-To: bmwg-admin@ietf.org
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.0.12
Precedence: bulk
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7bit

Kevin,

This work, in my opinion is clearly a benchmarking test, like any other
test that may measure benchmarks such as through throughput, latency,
frame loss rate, system recovery, reset etc. This effort will measure
similar benchmarks in the context of each of the configuration modules
proposed (control, data, management, security). The only differnce will
be that such benchmarking parameters will be gathered under a predefined
steady state.

For example, lets consider the routing protocol configuration set under
control plane module. Our measurement units for the ISIS routing
protocol include (Section 3.2.1.1) -

	   ISIS					Enabled/Disabled
	   ISIS-TE				Enabled/Disabled
	   Number of ISIS Adjacencies		Adjacencies
	   Number of ISIS Routes		Routes
	   Number of Nodes per Area		Nodes

Each of these measurement units can contribute to a becnhmark test. Lets
say we want to benchmark the max. number of ISIS routes. One way of
doing this is to run only ISIS on the DUT (and nothing else) and
determine the maximum number of ISIS routes supported with 0% frame loss
rate. Another way might be to do the same test, with a predefined steady
state consisting of all the other protocols that DUT may be required to
support (e.g. BGP, RSVP, etc) and gathering the same test data - how
many ISIS routes did the DUT support under these modified steady state
conditions at 0% frame loss rate? 

Regarding the concern about variability in test results, that by itself
is a benchmark in my opinion. If the DUT does exhibit widely variable
behavior, it is very concerning from an operational standpoint.
Predictability and a uniform response to stimulus is as important as
bare functionality.

In addition, even though it may be the case that increasing the number
of workload factors makes it harder to get a handle on what's going on,
the reality is that the real world works that way - most carriers are
running multiple protocols on their core devices, today. It is
concerning for me as a service provider if I hear that we, as a working
group cannot define how we can get a handle of what is happening on such
core routers even in a controlled environment.

Shankar.

-----Original Message-----
From: bmwg-admin@ietf.org [mailto:bmwg-admin@ietf.org] On Behalf Of
Kevin Dubray
Sent: Thursday, May 08, 2003 2:58 PM
To: bmwg@ietf.org
Cc: Perser, Jerry
Subject: Re: [bmwg] New WG Deliverable Proposal - Core Router Software
Accelerated Life Testing




Perser, Jerry wrote:
> Kevin,
> 
> I see Scott going down one of two paths: either this proposal is an 
> interesting software verification test, or a mixed benchmark.
> 
> I believe the title is worded such that it looks like a software 
> verification test.  If the path is an interesting test, it does fit in

> the BMWG.

I think the ADs might have something to say about the last premise.
"Interesting" isn't the sole entry condition for this WG.


> On the other hand, if this is a procedure for mixing previous 
> benchmarking RFCs, there is a desire to use it.  We have an RFC to 
> benchmark walking.  We have a benchmark for chewing gum.  But what 
> happens when you walk and chew gum?  We should focus on the metrics, 
> not if we swallow the gum.


It's my experience, that as you increase the number of workload factors
applied to a system under test, the harder it is to keep a handle on
what's going on.  Moreover, exacting replication of the workload is a
non-trivial exercise with a large number of workload factors.  Often,
responses have a wide degree fo variability - run the test once, get one
response; run it again, get another response. Sometimes, this is an 
artifact
of SUT behavior, but sometimes the test equipment behaves less than
uniformly, especially if we start mixing too many metaphors. :-)

Personally, I think this is a great test.   I also believe that the
proposed effort presents too many knobs to successfully produce a
dependable, uniform yardstick from which to gather comparative [core] 
router
reliability measurements.

> It would be valueable work if we expand mixing to more than unicast 
> and multicast frames.  The mcastm-12 I-D standardizes how to benchmark

> two classes of frames.  How about we expand this concept to other 
> benchmarks on a core router?

I'm sure there are many other factors that could be added to make the
operational profile more interesting; however, it also exacerbates the
difficulties highlighted above.

In lieu of this benchmarking forum, I believe the proposed topic is
better specified in a provider's validation test plan, where applicable
context can be specifically tailored to a provider's individual
operational requirements as well as its test equipments' capability.

-Kevin




_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg