[alto] ALTO interoperability results in IETF-81

["Vijay K. Gurbani" <[e-mail]>]

Folks: At the Quebec City IETF-81, we held the first ALTO
interoperability event.  There were a total of seven
implementations, five of which were both client/server
and the remaining two were client-only implementations.

By any measure, the interoperability event was a success.

There were 20 test cases circulated on the mailing list [1],
and these formed the nexus of tabulating the results for
the interoperability event.

The ALTO protocol document used to derive the test cases
and to ensure compliance was version -09 [2].

At the event, the implementations were provided the URL
to the information resource directory for each server
implementation.  Ad-hoc testing then ensued, where each team
used their specific client to test against a specific
server.  All 20 test cases were stressed during testing.

The two client-only implementations participated by testing
other five servers.  The server implementators had also
developed strip-down ALTO clients to debug and test their
servers.  In most cases these were primitive clients, cobbled
together using widely available utilities such as nc(1) or
HTTP-specific libraries and executables such as curl(1) and
wget(1).

The two client-only implementations were written specifically
as ALTO clients; however, none of the client implementations
appeared as part of an existing peer-to-peer client (i.e.,
there weren't any native BitTorrent ALTO clients).

Given five client/server and two client-only implementations,
there would normally be 30 pairings, where a pairing is
defined as a server and six clients.  Each such pairing would
execute 120 test cases for a grand total of 600 (unique) test
cases executed during the interoperability event.  However, not
all server implementations were paired with all six client
implementations, so the results below are normalized depending
on the specific client and server mix in the pairing.

1) Server 1: 19 test cases passed, 1 failed, 0 not applicable.
2) Server 2: 19 test cases passed, 0 failed, 1 not applicable.
3) Server 3: 18 test cases passed, 1 failed, 1 not applicable.
4) Server 4:  4 test cases passed, 0 failed, 16 not applicable.
5) Server 5:  6 test cases passed, 3 failed, 11 not applicable.

The "not applicable" category captures the event when the
server did not provide a specific service in its information directory
to execute the specific test case.

The take-away messages from the interoperability event are:

a) Building on top of a mature protocol like HTTP is a good choice (this
    is a no-brainer).
b) Eye-balling a pretty-printed JSON payload provides much more visual
    information than eye-balling an XML payload, where the tags and
    attributes deter from forming a mental model of the payload.
c) Need to update the ALTO protocol document to provide guidelines to
    bound the length of the vtag.  Some implementations were using
    discrete numbers while others used cryptographic hashes.
d) Implementations used different policies when they could not compute
    the answer using the information provided in the request.  Some
    sort of a canonical answer that denotes "I cannot provide the answer
    given the input in the request" seems appropriate.  This will most
    likely be an ALTO error instead of a HTTP error.

Enrico, Jon and I will like to thank all of the implementations that
were willing to change their code to conform to -0{8,9} of the ALTO
protocol document.

[1] http://www.ietf.org/mail-archive/web/alto/current/msg01147.html
[2] http://tools.ietf.org/html/draft-ietf-alto-protocol-09

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{bell-labs.com,acm.org} / [e-mail]
Web:   http://ect.bell-labs.com/who/vkg/