Re: [tap] RFC Status?

Leon Timmermans <fawaka@gmail.com> Wed, 17 September 2014 18:08 UTC

Return-Path: <fawaka@gmail.com>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7A2451A0864 for <tap@ietfa.amsl.com>; Wed, 17 Sep 2014 11:08:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Tpzzm985hzcO for <tap@ietfa.amsl.com>; Wed, 17 Sep 2014 11:08:44 -0700 (PDT)
Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 24CEA1A0745 for <tap@ietf.org>; Wed, 17 Sep 2014 11:08:10 -0700 (PDT)
Received: by mail-ig0-f171.google.com with SMTP id hn15so639003igb.10 for <tap@ietf.org>; Wed, 17 Sep 2014 11:08:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=OHGrb71ZOPf7BMDXMoA/SceCWQNwWiP+TK0hYd/iiHI=; b=pMJ285EyApP3L6ucAPe9v76ET9SKz1hR3bzBYM36NNPZ0OIvC9afvcu16VcqU8/YzQ Ya91qUjJ3MSHhOrwiVbMPMF9HoQlpRW74tuaN2ayzpaWhi1uOEMZ9tketQuPkCyxAuTH s0Y2eeR+L3Tx9UkPtQ5fSlX0HTUKN1WlQWJyam8nRTtxkWmYv6sbnIzviJ8jFRa3ZUKo rLmbGpMggfgRfxFURRn/z30dKYnGeeUXWbHBQQ/S7hhuTDgiKRBvoQG5IA1aJTq/crad AHVNw2D8qZkhh280rBv+pn0Xx+3Y6IXjNP56V8nBgE7hRMe2nlrwy5a2UjyV9fQoRQW/ WLiA==
X-Received: by 10.42.81.146 with SMTP id z18mr3394170ick.94.1410977289499; Wed, 17 Sep 2014 11:08:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.50.109.198 with HTTP; Wed, 17 Sep 2014 11:07:49 -0700 (PDT)
In-Reply-To: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com>
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com>
From: Leon Timmermans <fawaka@gmail.com>
Date: Wed, 17 Sep 2014 20:07:49 +0200
Message-ID: <CAHhgV8jr6ZnsfUkpFC4OL0AwRX-aen7v-7KjcN3e0_19s7steg@mail.gmail.com>
To: Andrew de Andrade <andrew@deandrade.com.br>
Content-Type: multipart/alternative; boundary=90e6ba6146fc2091dc050346c03b
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/NiyFTdRf9fu5yeyAP4Zby9VrPEY
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Sep 2014 18:08:46 -0000

On Mon, Aug 11, 2014 at 8:42 PM, Andrew de Andrade <andrew@deandrade.com.br>
wrote:

> I'm working on some testing libraries for JavaScript based on TAP.
> Specifically, I'm trying to explore making it possible for library authors
> to not only check the results of their TAP tests, but also map-reduce the
> TAP results of the dependents of their library to see how changes to their
> library impact the correctness of their dependents.
>
> e.g. I am the author of library A. I publish version 0.1.0. Another author
> includes my library as a dependency in library B and in library C and uses
> TAP to test those libraries. Given this scenario, it would be nice if I
> could, when modifying my library, run not only my tests, but also run the
> tests of all my dependents built with my previous version and my current
> version and compare how my changes impacted the "correctness" of their
> programs.
>
> Beyond the benefits to a community, helping authors know when they can
> likely safely upgrade their deps, this type of runner would also help in an
> corporate environment with lots of code re-use between teams.
>
> While thinking about this, I decided to go out and figure out if there is
> such a thing as nested TAP in the protocol specification. While searching
> for this, I came across this group, which is a great, but it appears there
> has been no activity here in a while and it looks like all the wiki pages
> with prior information about the state of the TAP protocol becoming an IETF
> RFC have disappeared.
>
> What's the current state of TAP? Is there still interest in taking this to
> RFC status? I found some previous discussions on the list about nested TAP.
> Did those make it into the specification in any way? If so, where can I
> find examples of correct nested tap results?
>
> Furthermore, is there a standardized set of test fixtures for the current
> version that any implementation can be tested against for correctness and
> performance? Having worked a bit with JSON schema, I found that one of the
> most useful tools was a standardized set of tests to check if a particular
> implementation conforms to the protocol standard. (see:
> https://github.com/json-schema/JSON-Schema-Test-Suite )
>
>
I've been recently writing a TAP Harness, and for subtests I followed what
Perl has been doing for years (just like TAP::Stream): have indented
subtests follow by a non-indented summary line. None of this is described
in detail anywhere though :-/, nor is the interaction with the barely
defined YAMLish well-explored AFAIK.

Leon