Re: [tap] RFC Status?

Ovid <> Wed, 17 September 2014 19:08 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id A97D81A06D1 for <>; Wed, 17 Sep 2014 12:08:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.649
X-Spam-Status: No, score=-3.649 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-1.652, SPF_PASS=-0.001, URIBL_DBL_ABUSE_REDIR=0.001, URIBL_DBL_REDIR=0.001] autolearn=ham
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id bTLLDJpLCLhq for <>; Wed, 17 Sep 2014 12:08:51 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 01A3B1A0AE0 for <>; Wed, 17 Sep 2014 12:08:50 -0700 (PDT)
Received: from [] by with NNFMP; 17 Sep 2014 19:08:50 -0000
Received: from [] by with NNFMP; 17 Sep 2014 19:08:49 -0000
Received: from [] by with NNFMP; 17 Sep 2014 19:08:49 -0000
X-Yahoo-Newman-Property: ymail-3
Received: (qmail 93164 invoked by uid 60001); 17 Sep 2014 19:08:49 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=s1024; t=1410980929; bh=EGY14F8BurvVNA0g/C0nhPQzUt1orSsAt9JuD21poTs=; h=References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=G1Qx1BlosrtqFy67pG2qwweNk5cJLaxHGt2PZApigsDWMRK7vF65+GzvOXuJfJrSwsck45eY1N1yPV69imSlQv/oAGqgrc/MctkHHSPv75nRdn8tFfY91RF/6jqAXBUDD9xgegrXjATwKsU4rUwvklQfVr4gnDIL8PWud4bO528=
X-YMail-OSG: 29cvnLcVM1njojHBpqCNmxrwgs875mgyk9SwM8CxfNVhVjl T3YzLruMkzgF_qvHd29MZrrzbt91x387EMf3OjpaqPv571SDVBhXu_u0gCUh C6BVVzG7gkPn1JIhM7n2kM9DVU4C85eTC0WdrFGJmMRE4.qI8KTTX7sWXctt HYy3kAW5Lh50B9AY1j2LckT_MsBdu4H3GofYJb5peEev_JotmhzvmulOi2DM CPO324dIntDfsBZJVhsqTJ_ngaMud14jwyZ_b63vfmtLoTHxNfNbw8xtXwAa n1H.FT7WGOEH5bT_woXO_2iwpK0rAYz5txIc6hBtzQSpV.uVTIAgs601aXa6 kcaZqdwmaumzTc6YD3PiP8QyfPhVCJ2JueTujFOQecq6ZIwPoYBWYIPjAmRc qfy9B9uikB24Z7Qq4l2gIObifZjjXx0d_NvsN0qwsh.hZjB6TAFABTbT9Qby s9NHKXRi6lqVL.rWsgIut9Jp7zrmH4SSPO4hOuGvPTeF56cCF9ctgvA7YAzH iBwlB1vZq2TSHtggx88Uy_yRAhVxh6__BNE06dlmizP7Xd5HDGafNftXYyHY 7ogq1YOHw5OXD4CtUf..UDOLwcL0djOvZeji4rkjKJZaMi4agFana_Xr3S7W 4W98n5lT4uD2bZWTA2e_iQ02FJiHC6o3BZmjkyuJDbs9R__HCz8ngi6F3.R6 3m7jaAiQkA2Ii_kp1UgRZzbs2NHM0NLnZhglGSn0F8H0ANagQk9scchwNEqV QsDTwXirtjCtVN1kXEHHEd6DpO3xcu1dcCwCtEWr5ovUyoOEgEKx8WANIe15 tfhmfb4zH5ktITHCKMIUxPrJ3EKCC.GlGPKwaeRu3acJVR_aOgQ8Z64j61XM 23R6EgoNRTRcBsrYT3sKIJpVefPiYgqS2QOfuz3PcQ1y1PHgoLNFCmcNq4EW pyGiLfOL5Ffc-
Received: from [] by via HTTP; Wed, 17 Sep 2014 12:08:49 PDT
X-Rocket-MIMEInfo: 002.001, QSBmZXcgbm90ZXMgb24gdGhpcy4KCkZvciBzdWJ0ZXN0cywgdGhleSBhcmUgaW5kZW50ZWQgZm91ciBzcGFjZXMuIEFsd2F5cyBmb3VyIHNwYWNlcy4gSWYgdGhleSBhcmUgaW5kZW50ZWQgZWlnaHQgc3BhY2VzLCB5b3Uga25vdyB0aGV5J3JlIGEgc3VidGVzdCBvZiBhIHN1YnRlc3QgKGFzc3VtaW5nIHRoZXJlJ3MgYW4gaW50ZXJ2ZW5pbmcgc3VidGVzdC4gVGhhdCBtZWFucyB0aGlzIGlzIHZhbGlkIFRBUCwgYnV0IHRoZXJlIGlzIG5vIHN1YnRlc3QgYmVjYXVzZSBpdCdzIGluZGVudGVkIHRvbyBmYXI6Cm8BMAEBAQE-
X-Mailer: YahooMailWebService/
References: <> <>
Message-ID: <>
Date: Wed, 17 Sep 2014 12:08:49 -0700
From: Ovid <>
To: Leon Timmermans <>, Andrew de Andrade <>
In-Reply-To: <>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="1181703601-1664221563-1410980929=:82809"
Cc: "" <>
Subject: Re: [tap] RFC Status?
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: Ovid <>
List-Id: Test Anything Protocol WG discussions <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 17 Sep 2014 19:08:56 -0000

A few notes on this.

For subtests, they are indented four spaces. Always four spaces. If they are indented eight spaces, you know they're a subtest of a subtest (assuming there's an intervening subtest. That means this is valid TAP, but there is no subtest because it's indented too far:
ok 1 - foo
>        ok 1 - bar indented too far
>ok 2 - bar passed
The "ok 1 - bar" line would be discarded as "unknown". 

So if, in a stream of TAP, you find something which would be valid tap if it were unindented by four spaces, it's a subtest. The trailing, unindented summary line must mirror the status of the subtest. Thus, the following is invalid:

ok 1 - foo
>    not ok 1 - bar indented too far
>ok 2 - bar passed
>Currently, Test::Harness treats the above as passing when it should not. I believe Schwern filed a bug for that. I know Andy Armstrong and I both took a quick swing at extending Test::Harness but discovered that it was more difficult than it appeared.

The grammar would be modified to look sort of like this (modifying

line           ::= (comment | test | subtest | unknown | bailout ) "\n"
>subtest        ::= subtest_line { subtest_line } test
>subtest_line   ::= '    ' line
That assumes we're OK with a recursive grammar. Also, the grammar doesn't really have a way of identifying the semantics of the trailing line on the subtest grammar line.

If you have other questions about subtests, I think I can give definitive answers.

As for the YAMLish structured diagnostics, those were never official, as far as I recall. There were several disputes, one of which was whether or not we should use YAML or JSON (I recall even XML being mentioned). Some argued that any structured diagnostic should be allowed, but I reject that firmly because that makes life hell for the parser. If you asked me what my stance is today, I would think that structured diagnostics should valid JSON, but that raises questions about whitespace issues, UTF-8, and so on. If we could agree on the format, then we'd need to agree on the keys in the structured diagnostic dictionary (and how to extend them).

IT consulting, training, international recruiting
Buy my book! -
Live and work overseas -

On Wednesday, 17 September 2014, 20:07, Leon Timmermans <> wrote:

>On Mon, Aug 11, 2014 at 8:42 PM, Andrew de Andrade <> wrote:
>I'm working on some testing libraries for JavaScript based on TAP. Specifically, I'm trying to explore making it possible for library authors to not only check the results of their TAP tests, but also map-reduce the TAP results of the dependents of their library to see how changes to their library impact the correctness of their dependents. 
>>e.g. I am the author of library A. I publish version 0.1.0. Another author includes my library as a dependency in library B and in library C and uses TAP to test those libraries. Given this scenario, it would be nice if I could, when modifying my library, run not only my tests, but also run the tests of all my dependents built with my previous version and my current version and compare how my changes impacted the "correctness" of their programs. 
>>Beyond the benefits to a community, helping authors know when they can likely safely upgrade their deps, this type of runner would also help in an corporate environment with lots of code re-use between teams. 
>>While thinking about this, I decided to go out and figure out if there is such a thing as nested TAP in the protocol specification. While searching for this, I came across this group, which is a great, but it appears there has been no activity here in a while and it looks like all the wiki pages with prior information about the state of the TAP protocol becoming an IETF RFC have disappeared. 
>>What's the current state of TAP? Is there still interest in taking this to RFC status? I found some previous discussions on the list about nested TAP. Did those make it into the specification in any way? If so, where can I find examples of correct nested tap results?
>>Furthermore, is there a standardized set of test fixtures for the current version that any implementation can be tested against for correctness and performance? Having worked a bit with JSON schema, I found that one of the most useful tools was a standardized set of tests to check if a particular implementation conforms to the protocol standard. (see: )
>I've been recently writing a TAP Harness, and for subtests I followed what Perl has been doing for years (just like TAP::Stream): have indented subtests follow by a non-indented summary line. None of this is described in detail anywhere though :-/, nor is the interaction with the barely defined YAMLish well-explored AFAIK.
>tap mailing list