[core] Preliminary Summary from CoAP#4 plugtest in London

Carsten Bormann <cabo@tzi.org> Sun, 09 March 2014 18:17 UTC

Return-Path: <cabo@tzi.org>
X-Original-To: core@ietfa.amsl.com
Delivered-To: core@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5C6221A028C for <core@ietfa.amsl.com>; Sun, 9 Mar 2014 11:17:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.951
X-Spam-Level:
X-Spam-Status: No, score=-0.951 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_DE=0.35, J_CHICKENPOX_42=0.6, SPF_HELO_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DJFElVzPKzHm for <core@ietfa.amsl.com>; Sun, 9 Mar 2014 11:17:04 -0700 (PDT)
Received: from informatik.uni-bremen.de (mailhost.informatik.uni-bremen.de [IPv6:2001:638:708:30c9::12]) by ietfa.amsl.com (Postfix) with ESMTP id D7E301A00DE for <core@ietf.org>; Sun, 9 Mar 2014 11:17:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at informatik.uni-bremen.de
Received: from smtp-fb3.informatik.uni-bremen.de (smtp-fb3.informatik.uni-bremen.de [134.102.224.120]) by informatik.uni-bremen.de (8.14.5/8.14.5) with ESMTP id s29IGql9001563 for <core@ietf.org>; Sun, 9 Mar 2014 19:16:52 +0100 (CET)
Received: from [10.109.75.237] (unknown [62.232.113.211]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp-fb3.informatik.uni-bremen.de (Postfix) with ESMTPSA id 92D5F231; Sun, 9 Mar 2014 19:16:46 +0100 (CET)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
From: Carsten Bormann <cabo@tzi.org>
Date: Sun, 9 Mar 2014 18:16:45 +0000
X-Mao-Original-Outgoing-Id: 416081805.747889-16e784c0717a716ed21c0532658b1edf
Content-Transfer-Encoding: quoted-printable
Message-Id: <07326969-B456-4E89-9417-5C07BDE8B3CA@tzi.org>
To: "core@ietf.org WG" <core@ietf.org>
X-Mailer: Apple Mail (2.1874)
Archived-At: http://mailarchive.ietf.org/arch/msg/core/p539wP2H0PNmekHFnlB2bJfXXZQ
Subject: [core] Preliminary Summary from CoAP#4 plugtest in London
X-BeenThere: core@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Constrained RESTful Environments \(CoRE\) Working Group list" <core.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/core>, <mailto:core-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/core/>
List-Post: <mailto:core@ietf.org>
List-Help: <mailto:core-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/core>, <mailto:core-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 09 Mar 2014 18:17:06 -0000

Here is an impression from the CoAP#4 plugtest that is ending today:

For the CoAP test suite, overall interoperability was 98.5 %, with
1346 interoperable out of 1367 tests completed.

The core-coap tests were interoperable at 99.3 % this time.

More spectacularly, the tests for Observe achieved 99.4 %.  This
result really reinforces shipping this specification to the IESG now.
I'm even more impressed that 48 hours after the Friday decision on
Observation cancellation, seven pairings (unfortunately, not everyone
had the local access to source code needed to do this) completed
testing the outcome, with 100 % interoperability.

Link-format fell back a little at 96.5 % (with problems mostly in
complex queries), and Block is at 95.1 %.  A remaining hotspot is
Block1 POST, which for over half of the pairings both sides had
implemented, but only achieved 70 % interoperability within this
group.  On the other hand, while only seven pairings had both
implemented Block1/Block2 POST, these achieved 100 % interoperability.

Thanks go to ETSI for running and to OMA for supporting the event, as
well as to all the participants for bringing lots of implementations
to the event.  After the IETF (and, for some of us, STRINT), this was
quite a marathon.

For those who just can’t stop testing (or have missed the event):

I have deployed the updates for CoAP#4 to http://coap.me, so you can
use coap://coap.me as a test CoAP server and use the web interface of
coap.me as a test CoAP client against your server.

The test descriptions for CoAP and DTLS can be found at
https://github.com/cabo/td-coap4 — these are not the official ETSI
document (which will be available soon) but the “source code” for
that.

Also don't miss testing with http://vs0.inf.ethz.ch, which complements
coap.me (e.g., it has an Observe implementation in the server, which
coap.me hasn't got around to).

If you want to arrange some timeslot for attended remote testing,
please don’t hesitate to ask me.

Grüße, Carsten