SNMP interoperability testin. Was: Re: snmpv2 Will it flood us or be slow

karl@mel-brooks.tgv.com Mon, 11 October 1993 21:09 UTC

Date: Mon, 11 Oct 93 12:49:48 PDT
Message-Id: <9310111949.AA10035@mel-brooks.empirical.com>
To: rdk@cc.gatech.edu
Cc: snmp@uu.psi.com
In-Reply-To: Bobby Krupczak's message of Mon, 11 Oct 93 14:51:16 EDT <199310111851.AA24423@morticia.cc.gatech.edu>
Subject: SNMP interoperability testin. Was: Re: snmpv2 Will it flood us or be slow
Reply-To: karl@empirical.com
Sender: ietf-archive-request@IETF.CNRI.Reston.VA.US
From: karl@mel-brooks.tgv.com
Repository: empirical.com
Originating-Client: mel-brooks

 > >I'm actually being swayed towards the other side by the workable
 > >reference implementations.  I fetched the ISODE and CMU implementations
 > >and have been doing interoperability experiments.  So far, everything
 > >seems to work very well...

One of things I've noticed after years of tying to make V1
implementations talk to one another is this:

Most so-called interoperability testing involves only an extremely
small portion of the possible range of protocol interactions.
Interoperability testing rarely is able to engender the end-of-range
situations that appear after equipment has been running for a long
time (for example enough time so that sysUpTime rolls over), or when
the network is going wacko.  Recently, you and I had the "opportunity"
to (mis-) configure a switching Ethernet hub and as a result we
accidently generated about 10 billion (that's a 32-bit number with the
high order bit set) of packets and we discovered that some managers
had trouble with numbers that big.  ('Twas fun seeing the Ethernet
LEDs glow so brightly.)

Mere "interoperability" testing ought not to give one the confidence
that the gear that thas been tested will work properly when all hell
is breaking loose on the network or when one adds a new manager that
does its queries in a slightly different pattern.

The kind of testing which will generate a degree of confidence is that
testing which is perverse and intended to exercise more than the most
common of interactions.  This usually requires special test agents and
managers which don't necessarily follow the rules, which easily
generate the limiting cases, and then exceed them.

I have heard of various people who have assembled some very clever
test programs.  (I wish I were at liberty to reveal them -- some of
the authors *are* on this list... hint, hint.)

What we need is a good, solid test facility which not only has lots of
field proven implementations, but also some specialized test engines
and some devilish "let's see if we can break this thing" minded
overseers.  (This, of course, needs to be coupled with a constructive,
"now that we broke it, how can we fix it" kind of attitude.)

BTW - The term "reference implementation" is misleading.  Such
implementations are merely early implementations worthy of no more
merit than any other except as a point of departure when trying to
resolve ambiguities in the specification.  I would hope the community
would stop using the term.

				--karl--

SNMP interoperability testin. Was: Re: snmpv2 Wil… karl