[alto] ALTO interoperability results in IETF-81

"Vijay K. Gurbani" <vkg@bell-labs.com> Mon, 08 August 2011 22:17 UTC

Return-Path: <vkg@bell-labs.com>
X-Original-To: alto@ietfa.amsl.com
Delivered-To: alto@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4A84721F8BEE for <alto@ietfa.amsl.com>; Mon, 8 Aug 2011 15:17:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.599
X-Spam-Level:
X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YMGeEllk7pjZ for <alto@ietfa.amsl.com>; Mon, 8 Aug 2011 15:17:19 -0700 (PDT)
Received: from ihemail3.lucent.com (ihemail3.lucent.com [135.245.0.37]) by ietfa.amsl.com (Postfix) with ESMTP id 8269F21F850E for <alto@ietf.org>; Mon, 8 Aug 2011 15:17:19 -0700 (PDT)
Received: from usnavsmail4.ndc.alcatel-lucent.com (usnavsmail4.ndc.alcatel-lucent.com [135.3.39.12]) by ihemail3.lucent.com (8.13.8/IER-o) with ESMTP id p78MHjiA004031 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for <alto@ietf.org>; Mon, 8 Aug 2011 17:17:45 -0500 (CDT)
Received: from umail.lucent.com (umail-ce2.ndc.lucent.com [135.3.40.63]) by usnavsmail4.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p78MHjPG006312 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for <alto@ietf.org>; Mon, 8 Aug 2011 17:17:45 -0500
Received: from shoonya.ih.lucent.com (shoonya.ih.lucent.com [135.185.238.235]) by umail.lucent.com (8.13.8/TPES) with ESMTP id p78MHjav027677 for <alto@ietf.org>; Mon, 8 Aug 2011 17:17:45 -0500 (CDT)
Message-ID: <4E4060F7.3050100@bell-labs.com>
Date: Mon, 08 Aug 2011 17:19:35 -0500
From: "Vijay K. Gurbani" <vkg@bell-labs.com>
Organization: Bell Laboratories, Alcatel-Lucent
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110621 Fedora/3.1.11-1.fc14 Lightning/1.0b2 Thunderbird/3.1.11
MIME-Version: 1.0
To: alto <alto@ietf.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.57 on 135.245.2.37
X-Scanned-By: MIMEDefang 2.64 on 135.3.39.12
Subject: [alto] ALTO interoperability results in IETF-81
X-BeenThere: alto@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Application-Layer Traffic Optimization \(alto\) WG mailing list" <alto.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/alto>, <mailto:alto-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/alto>
List-Post: <mailto:alto@ietf.org>
List-Help: <mailto:alto-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/alto>, <mailto:alto-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Aug 2011 22:17:20 -0000

Folks: At the Quebec City IETF-81, we held the first ALTO
interoperability event.  There were a total of seven
implementations, five of which were both client/server
and the remaining two were client-only implementations.

By any measure, the interoperability event was a success.

There were 20 test cases circulated on the mailing list [1],
and these formed the nexus of tabulating the results for
the interoperability event.

The ALTO protocol document used to derive the test cases
and to ensure compliance was version -09 [2].

At the event, the implementations were provided the URL
to the information resource directory for each server
implementation.  Ad-hoc testing then ensued, where each team
used their specific client to test against a specific
server.  All 20 test cases were stressed during testing.

The two client-only implementations participated by testing
other five servers.  The server implementators had also
developed strip-down ALTO clients to debug and test their
servers.  In most cases these were primitive clients, cobbled
together using widely available utilities such as nc(1) or
HTTP-specific libraries and executables such as curl(1) and
wget(1).

The two client-only implementations were written specifically
as ALTO clients; however, none of the client implementations
appeared as part of an existing peer-to-peer client (i.e.,
there weren't any native BitTorrent ALTO clients).

Given five client/server and two client-only implementations,
there would normally be 30 pairings, where a pairing is
defined as a server and six clients.  Each such pairing would
execute 120 test cases for a grand total of 600 (unique) test
cases executed during the interoperability event.  However, not
all server implementations were paired with all six client
implementations, so the results below are normalized depending
on the specific client and server mix in the pairing.

1) Server 1: 19 test cases passed, 1 failed, 0 not applicable.
2) Server 2: 19 test cases passed, 0 failed, 1 not applicable.
3) Server 3: 18 test cases passed, 1 failed, 1 not applicable.
4) Server 4:  4 test cases passed, 0 failed, 16 not applicable.
5) Server 5:  6 test cases passed, 3 failed, 11 not applicable.

The "not applicable" category captures the event when the
server did not provide a specific service in its information directory
to execute the specific test case.

The take-away messages from the interoperability event are:

a) Building on top of a mature protocol like HTTP is a good choice (this
    is a no-brainer).
b) Eye-balling a pretty-printed JSON payload provides much more visual
    information than eye-balling an XML payload, where the tags and
    attributes deter from forming a mental model of the payload.
c) Need to update the ALTO protocol document to provide guidelines to
    bound the length of the vtag.  Some implementations were using
    discrete numbers while others used cryptographic hashes.
d) Implementations used different policies when they could not compute
    the answer using the information provided in the request.  Some
    sort of a canonical answer that denotes "I cannot provide the answer
    given the input in the request" seems appropriate.  This will most
    likely be an ALTO error instead of a HTTP error.

Enrico, Jon and I will like to thank all of the implementations that
were willing to change their code to conform to -0{8,9} of the ALTO
protocol document.

[1] http://www.ietf.org/mail-archive/web/alto/current/msg01147.html
[2] http://tools.ietf.org/html/draft-ietf-alto-protocol-09

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{bell-labs.com,acm.org} / vijay.gurbani@alcatel-lucent.com
Web:   http://ect.bell-labs.com/who/vkg/