Re: [tap] Valids subtests (was: RFC Status?)

["Bruno P. Kinoshita" <[e-mail]>]

Hello Ovid, 

> In a test hackathon in Norway, we discussed creating a TAP document that would be a series of TAP snippets and how they should be parsed. The document would be YAML or something similar and should allow TAP implementors in various languages to parse the TAP snippets and compare their results to the document results. In other words, a language-agnostic test specification. Maybe now would be a good time to create one?

+1 for that, it would incrase compatibility between producer/consumers in different programming languages.

And we could derive from it a simple web page like JSON-LD has [1] and provide a more readable version as well.

Cheers, 
Bruno

[1] http://json-ld.org/test-suite/



>________________________________
> From: Ovid <[e-mail]>
>To: Andrew Rodland <[e-mail]> 
>Cc: "[e-mail]" <[e-mail]> 
>Sent: Friday, September 19, 2014 5:16 AM
>Subject: Re: [tap] Valids subtests (was: RFC Status?)
> 
>
>
>Hi Andrew,
>
>
>In that, we agreed that it was desirable that the determination of whether a line was junk/unknown wasn't context sensitive.
>
>
>I've tried searching for where I said that, but I can't find it. Can you provide me with the quote so I can understand where the confusion lies? I may very well be mistaken here.
>
>
>I fully admit that subtests were never considered in the original formulation of junk lines, but I think my description of this as being valid and the eight space indented line being an unknown (and therefore discarded) line is correct:
>
>
>> ok 1 - foo
>>        ok 1 - bar indented too far
>> ok 2 - bar passed
>
>
>The reasoning works like this: historically it has always been the case that ok/not ok status is bound to the beginning of the string and even a single space in front of ok/not ok invalid. With a subtest, we make an exception if an only if it is indented exactly four spaces (in the context of the containing scope) and is parseable as TAP. Otherwise, writing a parser could be a nightmare. Consider:
>
>
>> ok
>> not ok
>> ok
>> 1..3
>
>
>Most people aren't aware of this, but that's a perfectly valid (but failing) TAP stream. While it's not recommended, the test numbers have always been optional. Here's the relevant line from the TAP grammar:
>
>
> test ::= status positiveInteger? description? directive?
>
>
>Because TAP is designed to be both human-readable and computer-parseable, there exists the unfortunate problem that sometimes it's just a touch harder to parse than normal and some of the plans for it might make life more difficult. For example, if we followed your suggestion, we would then have this scenario:
>
>
>> ok
>>        ok this is working now
>> ok bar passed
>
>> 1..2
>
>
>As currently described (by my understanding), the above is valid because the "ok this is working now" line is indented too far and thus discarded. This makes parsing much simpler. However, you argue that the above is invalid because the "ok" line is valid TAP in an invalid context. That makes sense, but consider the plans for structured diagnostics:
>
>
>> not ok # TODO darn it!
>>     --- 
>
>>     file     : t/foo.t
>>     datetime : 2015-01-23 01:23:45
>>     have     : 7
>
>>     want     : 8
>>     comment  : |
>
>>                 We still have an off by one error and it's not
>>                 ok that we still haven't tracked it down
>>     ...
>> ok bar passed
>
>> 1..2
>
>
>Parsers are not required to parse structured diagnostics (this allows older parsers to ignore them), but that last line of the comment above could be considered valid TAP in an invalid context and result in a parse error, even though the above is perfectly fine.
>
>
>The one thing I really don't want to see is for TAP to go down the YAML road of having a mind-bogglingly huge spec to cover all sorts of corner cases. This will block TAP adoption and make it much harder for everyone to get right. However, if we are strict about the four space rule and discarding unknown lines, any parser not compliant with that rule can (sort of) be ignored, but that means better communication about requirements.
>
>
>In a test hackathon in Norway, we discussed creating a TAP document that would be a series of TAP snippets and how they should be parsed. The document would be YAML or something similar and should allow TAP implementors in various languages to parse the TAP snippets and compare their results to the document results. In other words, a language-agnostic test specification. Maybe now would be a good time to create one?
>
>
>Best,
>Ovid
>--
>IT consulting, training, international recruiting
>     
  http://www.allaroundtheworld.fr/.
>Buy my book! - http://bit.ly/beginning_perl
>Live and work overseas - http://www.overseas-exile.com/
>
>
>
>On Friday, 19 September 2014, 5:58, Andrew Rodland <[e-mail]> wrote:
> 
>
>>
>>
>>> A few notes on this.
>>>
>>> For subtests, they are indented four spaces. Always four spaces. If they are
>>> indented eight spaces, you know they're a subtest of a subtest (assuming
>>> there's an intervening subtest. That means this is valid TAP, but there is
>>> no subtest because it's indented too far:
>>>
>>> ok 1 - foo
>>>     
    ok 1 - bar indented too far
>>> ok 2 - bar passed
>>>
>>> The "ok 1 - bar" line would be discarded as "unknown".
>>
>>I'm not trying to argue against established behavior, but I'd
 like to
>>point out that this disagrees with what you expressed in the previous
>>thread "More uncertainty about junk lines". In that, we agreed that it
>>was desirable that the determination of whether a line was
>>junk/unknown wasn't context sensitive. If it's potentially valid TAP
>>in any context, it's interpreted as that, and if that causes the TAP
>>document to be invalid, then so be it. If that holds true, then this
>>snippet is an error; it has a result in a subsubtest that lacks a plan
>>or a containing subtest.
>>
>>
>>Andrew
>>
>>
>>
>
>_______________________________________________
>tap mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/tap
>
>
>