Re: [tap] Valids subtests (was: RFC Status?)

Ovid <> Fri, 19 September 2014 08:17 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id 00F9D1A0016 for <>; Fri, 19 Sep 2014 01:17:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -0.949
X-Spam-Status: No, score=-0.949 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-1.652, SPF_PASS=-0.001, URIBL_DBL_ABUSE_REDIR=0.001, URIBL_DBL_REDIR=0.001] autolearn=ham
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 22fmFQLm6K_O for <>; Fri, 19 Sep 2014 01:17:00 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 802951A001A for <>; Fri, 19 Sep 2014 01:17:00 -0700 (PDT)
Received: from [] by with NNFMP; 19 Sep 2014 08:16:59 -0000
Received: from [] by with NNFMP; 19 Sep 2014 08:16:59 -0000
Received: from [] by with NNFMP; 19 Sep 2014 08:16:59 -0000
X-Yahoo-Newman-Property: ymail-3
Received: (qmail 22689 invoked by uid 60001); 19 Sep 2014 08:16:59 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=s1024; t=1411114619; bh=4SJ5hoDcer2TbmpfUmZ30DrbEiP8mzaz5vMyPSFT0h8=; h=References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=o6G/Zit05L6K8VPHzuiHds+sQtLo22gNQXoCtPjN9v8tcdyPILIhFvdayRReTthY5dHkEUB9AVmkVPR59wApnDIDGbBZyKpNfYe4P4N8iqDzPnyZWoyEiZ3g1BuMRHJuw4OCZNIg7ZeYMM5byBnHizj3Hscw8REMoiRFOBtjdpQ=
X-YMail-OSG: xOIQbaMVM1nJ0Xpg2_MmBAHx9IZO_5.IifZP4i6AZiWU31N hMy4ydrNIcHn5eydrkmCjlCrT.VcQrOT0P3RJsXEo6a6tbtJmxcmZHWJEChS U2KjsHFfcQVb9QNufjxSSxkJOy6kNSPM0TJ1G0GkB.OufDQxIqfJ3ool0BkK Dwxy_0NEAXZ7a0zqUCHKcWVikK7Ge5ACYBNYF8Uvq5PxdhkW4ToMffP2WFHr 95hX3kiVF95AjH9qJbYFSTK5YFIkXCZsc0_zzOmTdvDTQjC6ZSnRUI14fLPc k2TYnx7NVaxsq8aQfkqFA6K9Pg1k0HZ3txGMaoYopOt2nmcW68ZBNImOjpJO kqZbbW0GuAMOI.YjCNPH1BtIEwHhMSs40yaYYGsY5CxngMFJzCAiYvymzGiF tnr88uusaQscItdatbOSFAWaigTRXqiaepktNUfHDmfxpzWwVdWhrOWI_EAH 5awqjDetbh8VoSxzuhTcdgXT.GOsrE9cX5J_e7C1LfcokQwg_GAuV1hqy2A7 pjNpON9rfxJ_zO7lMZZPbVJ_y0quYfogMAQhGt.jW2RpjG3Kk.Y9SnIv.Qgn WIsyWjMIM3.hyUvHlMZrvZMg8dsFJi30UtheTt_.E0uNkqHZVw6Y9ut8pciO miGm5wDmoHXlnTOfdKvThwzdOuROa4p8tknooFtfbhPIImi2utnw5S639mW3 C1gORQFFInWUmd1GHNsixjPlCZwLsUXoJi_C14QDCsUWzICgCOHs5LUVlCm. Qq.gvdUFQB4Tzo.AKoNqf_wn7bg.Lcd5_woXPaBRN2iwI3ldqYkVDONdvmvF dx4GD70O2zTVN3X00Fk1BWDw_gNPkwIEkvMaCtiAS63J_25ly5mcI14t6j3i _4tl5hp9rvSonEevfxA--
Received: from [] by via HTTP; Fri, 19 Sep 2014 01:16:59 PDT
X-Rocket-MIMEInfo: 002.001, SGkgQW5kcmV3LAoKSW4gdGhhdCwgd2UgYWdyZWVkIHRoYXQgaXQgd2FzIGRlc2lyYWJsZSB0aGF0IHRoZSBkZXRlcm1pbmF0aW9uIG9mIHdoZXRoZXIgYSBsaW5lIHdhcyBqdW5rL3Vua25vd24gd2Fzbid0IGNvbnRleHQgc2Vuc2l0aXZlLgoKSSd2ZSB0cmllZCBzZWFyY2hpbmcgZm9yIHdoZXJlIEkgc2FpZCB0aGF0LCBidXQgSSBjYW4ndCBmaW5kIGl0LiBDYW4geW91IHByb3ZpZGUgbWUgd2l0aCB0aGUgcXVvdGUgc28gSSBjYW4gdW5kZXJzdGFuZCB3aGVyZSB0aGUgY29uZnVzaW9uIGxpZXM_IEkgbWF5IHYBMAEBAQE-
X-Mailer: YahooMailWebService/
References: <> <> <> <> <>
Message-ID: <>
Date: Fri, 19 Sep 2014 01:16:59 -0700
From: Ovid <>
To: Andrew Rodland <>
In-Reply-To: <>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="1966909624-73427268-1411114619=:77920"
Cc: "" <>
Subject: Re: [tap] Valids subtests (was: RFC Status?)
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: Ovid <>
List-Id: Test Anything Protocol WG discussions <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 19 Sep 2014 08:17:06 -0000

Hi Andrew,

In that, we agreed that it was desirable that the determination of whether a line was junk/unknown wasn't context sensitive.

I've tried searching for where I said that, but I can't find it. Can you provide me with the quote so I can understand where the confusion lies? I may very well be mistaken here.

I fully admit that subtests were never considered in the original formulation of junk lines, but I think my description of this as being valid and the eight space indented line being an unknown (and therefore discarded) line is correct:

> ok 1 - foo
>        ok 1 - bar indented too far
> ok 2 - bar passed

The reasoning works like this: historically it has always been the case that ok/not ok status is bound to the beginning of the string and even a single space in front of ok/not ok invalid. With a subtest, we make an exception if an only if it is indented exactly four spaces (in the context of the containing scope) and is parseable as TAP. Otherwise, writing a parser could be a nightmare. Consider:

> ok
> not ok
> ok
> 1..3

Most people aren't aware of this, but that's a perfectly valid (but failing) TAP stream. While it's not recommended, the test numbers have always been optional. Here's the relevant line from the TAP grammar:

 test ::= status positiveInteger? description? directive?

Because TAP is designed to be both human-readable and computer-parseable, there exists the unfortunate problem that sometimes it's just a touch harder to parse than normal and some of the plans for it might make life more difficult. For example, if we followed your suggestion, we would then have this scenario:

> ok
>        ok this is working now
> ok bar passed

> 1..2

As currently described (by my understanding), the above is valid because the "ok this is working now" line is indented too far and thus discarded. This makes parsing much simpler. However, you argue that the above is invalid because the "ok" line is valid TAP in an invalid context. That makes sense, but consider the plans for structured diagnostics:

> not ok # TODO darn it!
>     --- 

>     file     : t/foo.t
>     datetime : 2015-01-23 01:23:45
>     have     : 7

>     want     : 8
>     comment  : |

>                 We still have an off by one error and it's not
>                 ok that we still haven't tracked it down
>     ...
> ok bar passed

> 1..2

Parsers are not required to parse structured diagnostics (this allows older parsers to ignore them), but that last line of the comment above could be considered valid TAP in an invalid context and result in a parse error, even though the above is perfectly fine.

The one thing I really don't want to see is for TAP to go down the YAML road of having a mind-bogglingly huge spec to cover all sorts of corner cases. This will block TAP adoption and make it much harder for everyone to get right. However, if we are strict about the four space rule and discarding unknown lines, any parser not compliant with that rule can (sort of) be ignored, but that means better communication about requirements.

In a test hackathon in Norway, we discussed creating a TAP document that would be a series of TAP snippets and how they should be parsed. The document would be YAML or something similar and should allow TAP implementors in various languages to parse the TAP snippets and compare their results to the document results. In other words, a language-agnostic test specification. Maybe now would be a good time to create one?

IT consulting, training, international recruiting
Buy my book! -
Live and work overseas -

On Friday, 19 September 2014, 5:58, Andrew Rodland <> wrote:

>> A few notes on this.
>> For subtests, they are indented four spaces. Always four spaces. If they are
>> indented eight spaces, you know they're a subtest of a subtest (assuming
>> there's an intervening subtest. That means this is valid TAP, but there is
>> no subtest because it's indented too far:
>> ok 1 - foo
>>         ok 1 - bar indented too far
>> ok 2 - bar passed
>> The "ok 1 - bar" line would be discarded as "unknown".
>I'm not trying to argue against established behavior, but I'd
 like to
>point out that this disagrees with what you expressed in the previous
>thread "More uncertainty about junk lines". In that, we agreed that it
>was desirable that the determination of whether a line was
>junk/unknown wasn't context sensitive. If it's potentially valid TAP
>in any context, it's interpreted as that, and if that causes the TAP
>document to be invalid, then so be it. If that holds true, then this
>snippet is an error; it has a result in a subsubtest that lacks a plan
>or a containing subtest.