Re: benchmarks ... (was: Re: the purpose of fddi mailing list ...

[Vernon Schryver <[e-mail]>]

> From: [e-mail] (Michael Yip)
> To: [e-mail]
> Subject: benchmarks ... (was: Re: the purpose of fddi mailing list ...
> 
> Can anyone tell me if UNH, which is going to perform the interop test
> on different vendors, will also perform some kind of benchmark type of
> performance testing on the equipments also?
> 
> As a suggestion, if we can come up with certain benchmarks or series of
> benchmarks, then UNH can perform the testing and let everyone know what
> the AVERAGE performance of the equipments.  If the vendor wishes to make
> the benchmark result for that piece of equipment public, then we can 
> start gathering more and more benchmarks...  Just a thought.
> 
> Is UNH listening?
> 
> -- Mike Yip
>    [e-mail]


The arithmetic average of the highest and lowest numbers I have heard would
not be interesting.  A median would not be interesting, since it would be
dominated by the large number of PC implementations.  Other values
such as "speed of routers costing $40,000-$80,000 and having 4 FDDI and 4
ethernet connections" probably be equivalent to publishing the numbers for
individuals.

Not all vendors belong to either of the two organizations that are in the
business of helping development and selling Good Housekeeping Seals of
approval.

Either organization might be helpful in developing and releasing a set of
public domain benchmarks.  Such a set of benchmarks would have to be
appropriate to all compeating technologies to be useful to customers.
Saying Acme's TCP/FDDI does 89.7 MB on the UNHANTC benchmark does not help
a customer deciding whether to buy Acme FDDI or the AAA HPPI boards.  It
would have to cover at least ethernet, HPPI, and ULTRANET.  Some customers
might need comparisons with Hyperchannel and XTP.

An external benchmark would be good for routers and bridges.  The recent
question was about host implementations.  It is not trivial write a
benchmark that runs on the target machine itself.  Such a benchmark would
have to be portable among the several flavors UNIX operating systems (SVR3,
SVR4, STREAMS, sockets, 4.2BSD, 4.3BSD), as well as the non-UNIX or
UNIX-compatible systems including DOS and the real-time systems.

Something like the SPEC mechanism might work someday, when all operating
systems have similar interfases.

BRL's familiar if not very pretty, benchmark, ttcp, (and presumably its
three or four common offspring) is portable among socket based systems.
Its numbers are the ones commonly quoted (ignoring FTP numbers).  Source
for ttcp is available from brl.mil and other places by anonymous FTP.  More
than one vendor is shipping ttcp objects and I think more than one will
soon be shippig ttcp source with their other products.


vjs