Re: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing

Perser, Jerry wrote:
> Kevin,
> 
> I see Scott going down one of two paths: either this proposal is an
> interesting software verification test, or a mixed benchmark.
> 
> I believe the title is worded such that it looks like a software
> verification test.  If the path is an interesting test, it does fit in the
> BMWG.

I think the ADs might have something to say about the last premise.
"Interesting" isn't the sole entry condition for this WG.

> On the other hand, if this is a procedure for mixing previous benchmarking
> RFCs, there is a desire to use it.  We have an RFC to benchmark walking.  We
> have a benchmark for chewing gum.  But what happens when you walk and chew
> gum?  We should focus on the metrics, not if we swallow the gum.

It's my experience, that as you increase the number of workload factors
applied to a system under test, the harder it is to keep a handle on
what's going on.  Moreover, exacting replication of the workload is a
non-trivial exercise with a large number of workload factors.  Often,
responses have a wide degree fo variability - run the test once, get one
response; run it again, get another response. Sometimes, this is an 
artifact
of SUT behavior, but sometimes the test equipment behaves less than
uniformly, especially if we start mixing too many metaphors. :-)

Personally, I think this is a great test.   I also believe that the
proposed effort presents too many knobs to successfully produce a
dependable, uniform yardstick from which to gather comparative [core] 
router
reliability measurements.

> It would be valueable work if we expand mixing to more than unicast and
> multicast frames.  The mcastm-12 I-D standardizes how to benchmark two
> classes of frames.  How about we expand this concept to other benchmarks on
> a core router?

I'm sure there are many other factors that could be added to make the
operational profile more interesting; however, it also exacerbates the
difficulties highlighted above.

In lieu of this benchmarking forum, I believe the proposed topic is
better specified in a provider's validation test plan, where applicable
context can be specifically tailored to a provider's individual
operational requirements as well as its test equipments' capability.

-Kevin

_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg