Re: [tap] W3C Evaluation and Report Language (EARL)

Leon Timmermans <fawaka@gmail.com> Wed, 09 November 2011 20:50 UTC

Return-Path: <fawaka@gmail.com>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 889361F0C7F for <tap@ietfa.amsl.com>; Wed, 9 Nov 2011 12:50:02 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.599
X-Spam-Level:
X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yFSPJCNjgGIU for <tap@ietfa.amsl.com>; Wed, 9 Nov 2011 12:50:02 -0800 (PST)
Received: from mail-vx0-f172.google.com (mail-vx0-f172.google.com [209.85.220.172]) by ietfa.amsl.com (Postfix) with ESMTP id E0C351F0C74 for <tap@ietf.org>; Wed, 9 Nov 2011 12:50:01 -0800 (PST)
Received: by vcbfk1 with SMTP id fk1so2045566vcb.31 for <tap@ietf.org>; Wed, 09 Nov 2011 12:50:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=4RZIA55Kb0QBESxeBvpCu/vEUH+L1Wy9S9mX2rFfVsk=; b=EQhRvKuAnGt1o9cesbTVcIMhtB3o6duwykb3SGh8oaQaMAcmctIUEirNulCf7ZUfcV 0QS4HqxdojJnLYLa0xQngC2ZeNKr5gkKIvHZdpezQnlGsKdVVpnATb10fjHq4/8B60CM 1dTj3jaM0jHURhUo9ZFHch1msLZK0pQsO8mNs=
Received: by 10.52.173.211 with SMTP id bm19mr7107093vdc.115.1320871801123; Wed, 09 Nov 2011 12:50:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.52.115.134 with HTTP; Wed, 9 Nov 2011 12:49:40 -0800 (PST)
In-Reply-To: <87d3d1qc0v.fsf@renormalist.net>
References: <6.2.5.6.2.20111109131023.02af29c0@esat.kuleuven.be> <87d3d1qc0v.fsf@renormalist.net>
From: Leon Timmermans <fawaka@gmail.com>
Date: Wed, 9 Nov 2011 21:49:40 +0100
Message-ID: <CAHhgV8h9-W5h8gNQGVv=q4kiZ38KvvJ2gedxcNsrnc1CdAJDJA@mail.gmail.com>
To: Steffen Schwigon <ss5@renormalist.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: tap@ietf.org, Shadi Abou-Zahra <shadi@w3.org>
Subject: Re: [tap] W3C Evaluation and Report Language (EARL)
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Nov 2011 20:50:02 -0000

On Wed, Nov 9, 2011 at 9:25 PM, Steffen Schwigon <ss5@renormalist.net> wrote:
> - EARL is similar to other W3C specs in respect to specifying a
>  comprehensive snapshot of known existing topics. For example, it
>  particularly covers all known HTTP methods (POST, GET, PUT, …). That
>  enables it to build tools on top of it that sematically “know” what
>  the document is about.
>
> - TAP in contrast is about specifying test results, really just the
>  *result* focus without hard specification of the tested topic, i.e., a
>  single test has a “description”, so someone reading it knows what it
>  is about but that part does not have a specification.
>
>  For instance, a test about a HTTP method could have any description
>  from “POST” to “that strange other method that I never remember but
>  always use when GET is not sufficient”.
>
>
> See [1] for some related discussion of this aspect.
>
> In this respect I think TAP is more like your RDF with some extensions
> from EARL to describe test success.
>
> That makes the use-cases of TAP and EARL a bit different:
>
> - TAP allows to be produced by anything simple without toolchain
>  support, like embedded devices with nothing but a “print” function,
>  but you can not *sematically* evaluate results.
>
> - EARL seems to require more heavy toolchain support to produce but
>  allows more semantic result evaluation.

I agree. To me, EARL makes sense in a context that heavy into semantic
web and triplestores, but the toolchain would be too heavy for a lot
of usecases.

> Converting TAP to EARL is difficult.
> Converting EARL to TAP is easy.

Making valid EARL out of TAP shouldn't be difficult. Useful EARL on
the other hand

> On the evaluation of TAP I can point to TAP::DOM and Data::DPath, which
> provide a more structured approach to evaluate test results, see my “TAP
> Juggling” slides[2], page 30ff.

Interesting…

Leon