Re: [newtrk] RE: Interoperability Testing Format

[Bruce Lilly <[e-mail]>]

On Mon August 8 2005 13:15, Larry Masinter wrote:

> It was pointed out that: 
>  
> http://www.ietf.org/internet-drafts/draft-ietf-idr-bgp-implementation-02.txt
> 
> 
> might be a "bad example" or at least a difficult case.

One problem is that it doesn't satisfy the requirements for advancing
the specification to Draft level; there is apparently only one (rather
than the required two independent) implementation which meets the
specification ("One vendor (NextHop) indicated "O" matching the
specification").

At least the document is arranged in an item-by-item manner which
facilitates identifying specification items which either result in
a failure to have the necessary two implementations or which must be
removed from the specification (BCP 9 sect. 4.1.2 2nd para.) for
advancement.

http://www.ietf.org/IESG/Implementations/rfc-2476-and-draft-gellens-submit-bis-implementation.txt
is a counterexample; it is poorly organized such that it is quite
difficult to determine that there are not in fact two implementations
which fully meet the specification as required by BCP 9.

Worse still is
http://www.ietf.org/IESG/Implementations/RFC2234-implementation-report.txt
which reads like a marketing blurb, contains factual errors, does not
list the specification items required, and lists two implementations,
neither of which meets the specification.

> I'd go for a flat structure as a list of features.

I'd settle for a summary of features something like that provided
at the end of each major section of RFC 1123, though strictly
speaking only the items affecting interoperability (i.e. not
"MAY", "SHOULD", "SHOULD NOT" items) are necessary.  That is, a
brief summary of the specification item and a reference to the
full specification text would be useful.

I'd prefer a formal implementation report submitted as documentation
for advancement to list the two independent implementations which
ostensibly meet all of the requirements, but I could accept Larry's
idea of separate per-implementation reports (however, in that case
it is important that each report contain all of the specification
items in the same order and format so that they can be easily
compared)[1].  Ideally, a matrix listing the specification items as rows
and the implementations as columns would be easiest to examine (are
there two columns that meet all specification items).

---------
1. But not XML.  It's not easily human-readable.  Plain text a la
   RFC 1123 is *greatly* preferable.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html