Re: [tap] W3C Evaluation and Report Language (EARL)

Steffen Schwigon <ss5@renormalist.net> Wed, 09 November 2011 20:25 UTC

Return-Path: <ss5@renormalist.net>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 34E7B1F0C53 for <tap@ietfa.amsl.com>; Wed, 9 Nov 2011 12:25:07 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_43=0.6]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Qts+DoMi8b9w for <tap@ietfa.amsl.com>; Wed, 9 Nov 2011 12:25:06 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by ietfa.amsl.com (Postfix) with ESMTP id 34D211F0C7F for <tap@ietf.org>; Wed, 9 Nov 2011 12:25:05 -0800 (PST)
Received: by h132974.serverkompetenz.net (Postfix, from userid 1000) id 4B4583F23ED; Wed, 9 Nov 2011 20:25:04 +0000 (GMT)
From: Steffen Schwigon <ss5@renormalist.net>
To: Christophe Strobbe <christophe.strobbe@esat.kuleuven.be>
References: <6.2.5.6.2.20111109131023.02af29c0@esat.kuleuven.be>
Date: Wed, 09 Nov 2011 20:25:04 +0000
In-Reply-To: <6.2.5.6.2.20111109131023.02af29c0@esat.kuleuven.be> (Christophe Strobbe's message of "Wed, 09 Nov 2011 14:23:18 +0100")
Message-ID: <87d3d1qc0v.fsf@renormalist.net>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Cc: tap@ietf.org, Shadi Abou-Zahra <shadi@w3.org>
Subject: Re: [tap] W3C Evaluation and Report Language (EARL)
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Nov 2011 20:25:07 -0000

Christophe Strobbe <christophe.strobbe@esat.kuleuven.be> writes:
> Report Language (EARL) and a few related specifications. EARL's core
> use case is reporting the results of accessibility evaluations of
> websites (i.e. accessibility for persons with disabilities), but the
> language itself is generic, so it can also be used in other
> contexts. The language is based on RDF;
> […]
> During our last call for comments, one of the reviewers asked the
> working group if EARL duplicates TAP's efforts, or vice versa. The
> working group thinks that this is not the case; we think that EARL
> could be an alternative report format for TAP if a TAP consumer could
> be written that produces EARL. For this reason, we thought it would be
> interesting to contact you and to make sure we are aware of each
> other's work.

Thanks for sync'ing this back to us. I just skimmed through the specs
and it was indeed interesting. As far as I understand from my (very
short) skimming I think it's not that many duplication of effort as the
main difference is a philosophical one.

- EARL is similar to other W3C specs in respect to specifying a
  comprehensive snapshot of known existing topics. For example, it
  particularly covers all known HTTP methods (POST, GET, PUT, …). That
  enables it to build tools on top of it that sematically “know” what
  the document is about.

- TAP in contrast is about specifying test results, really just the
  *result* focus without hard specification of the tested topic, i.e., a
  single test has a “description”, so someone reading it knows what it
  is about but that part does not have a specification. 

  For instance, a test about a HTTP method could have any description
  from “POST” to “that strange other method that I never remember but
  always use when GET is not sufficient”.


See [1] for some related discussion of this aspect.

In this respect I think TAP is more like your RDF with some extensions
from EARL to describe test success.

That makes the use-cases of TAP and EARL a bit different: 

- TAP allows to be produced by anything simple without toolchain
  support, like embedded devices with nothing but a “print” function,
  but you can not *sematically* evaluate results.

- EARL seems to require more heavy toolchain support to produce but
  allows more semantic result evaluation.

Converting TAP to EARL is difficult. 
Converting EARL to TAP is easy.

On the evaluation of TAP I can point to TAP::DOM and Data::DPath, which
provide a more structured approach to evaluate test results, see my “TAP
Juggling” slides[2], page 30ff.

Kind regards,
Steffen


Footnotes: 
[1]  http://grokbase.com/p/perl.org/qa/2008/04/re-tap-l-user-supplied-yaml-diagnostic-keys-descriptive-version/11ymnpm2765ztojoinznq2lz5674

[2]  http://www.amd64.org/fileadmin/user_upload/pub/yapc_eu_2011_tapjuggling.pdf

-- 
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers <http://dresden-pm.org/>