Re: [tap] Valids subtests (was: RFC Status?)

[Andrew Rodland <[e-mail]>]

On Fri, Sep 19, 2014 at 4:16 AM, Ovid <[e-mail]> wrote:
> Hi Andrew,
>
> In that, we agreed that it was desirable that the determination of whether a
> line was junk/unknown wasn't context sensitive.
>
> I've tried searching for where I said that, but I can't find it. Can you
> provide me with the quote so I can understand where the confusion lies? I
> may very well be mistaken here.
>

=== begin quote (http://www.ietf.org/mail-archive/web/tap/current/msg00480.html)

That's a very annoying edge case you have there, but I would argue
that anything which *could* be parsed as a TAP line MUST be parsed as
such, thus rendering the above invalid. Otherwise, we'd have no idea
if you were spitting out bad TAP or something else was spitting out
junk which happened to look like TAP.

Thus:

* If something *can* be parsed as a TAP line, do so, even if it
renders the TAP document invalid.
* If something *can't* be parsed as a TAP line, ignore it.

=== end quote

> I fully admit that subtests were never considered in the original
> formulation of junk lines, but I think my description of this as being valid
> and the eight space indented line being an unknown (and therefore discarded)
> line is correct:
>
>> ok 1 - foo
>>        ok 1 - bar indented too far
>> ok 2 - bar passed
>
> The reasoning works like this: historically it has always been the case that
> ok/not ok status is bound to the beginning of the string and even a single
> space in front of ok/not ok invalid. With a subtest, we make an exception if
> an only if it is indented exactly four spaces (in the context of the
> containing scope) and is parseable as TAP. Otherwise, writing a parser could
> be a nightmare. Consider:
>
>> ok
>> not ok
>> ok
>> 1..3
>
> Most people aren't aware of this, but that's a perfectly valid (but failing)
> TAP stream. While it's not recommended, the test numbers have always been
> optional. Here's the relevant line from the TAP grammar:
>
>  test ::= status positiveInteger? description? directive?
>
> Because TAP is designed to be both human-readable and computer-parseable,
> there exists the unfortunate problem that sometimes it's just a touch harder
> to parse than normal and some of the plans for it might make life more
> difficult. For example, if we followed your suggestion, we would then have
> this scenario:
>
>> ok
>>        ok this is working now
>> ok bar passed
>> 1..2
>
> As currently described (by my understanding), the above is valid because the
> "ok this is working now" line is indented too far and thus discarded. This
> makes parsing much simpler. However, you argue that the above is invalid
> because the "ok" line is valid TAP in an invalid context. That makes sense,
> but consider the plans for structured diagnostics:
>
>> not ok # TODO darn it!
>>     ---
>>     file     : t/foo.t
>>     datetime : 2015-01-23 01:23:45
>>     have     : 7
>>     want     : 8
>>     comment  : |
>>                 We still have an off by one error and it's not
>>                 ok that we still haven't tracked it down
>>     ...
>> ok bar passed
>> 1..2
>
> Parsers are not required to parse structured diagnostics (this allows older
> parsers to ignore them), but that last line of the comment above could be
> considered valid TAP in an invalid context and result in a parse error, even
> though the above is perfectly fine.
>

Yes, I see the conflict here. I could suggest that structured
diagnostics be indented *two* spaces relative to the test that they
annotate, which would eliminate any potential for confusion since a
valid test is going to be indented 4n spaces. I'm not sure if that's
too esoteric.