Re: [tap] "ok 1 description" vs "ok 1 - description"

[Michael G Schwern <[e-mail]>]

Steffen Schwigon wrote:
> Paul Johnson <[e-mail]> writes:
>> On Sat, Jan 24, 2009 at 01:55:06AM -0800, Ovid wrote:
>>> ----- Original Message ----
>>>
>>>> From: Michael G Schwern <[e-mail]>
>>>> Test-Result     = Status [SP Test-Number] [SP Description]
>>>>                    [SP "#" SP Directive [SP Reason]] EOL
>>>>
>>>> to
>>>>
>>>> Test-Result     = Status [SP Test-Number] [SP -] [SP Description]
>>>>                    [SP "#" SP Directive [SP Reason]] EOL
>>>>
>>>> The dash is simply ignored if it's there.
>>>>
>>>> Yay or nay?
>>> So everyone who's written a TAP producer in another language and
>>> didn't see fit to add that dash will now be producing invalid TAP?
>> I don't think so, since the dash is optional.

Exactly, the dash is optional so if you leave it out that's fine.  Sorry for
not making that abundantly clear, I forget not everyone can read ABNF.

These two lines are semantically equivalent:

ok 1 foo
ok 1 - foo

They are both passing test number 1 with a description of "foo".


>> The only real
>> complication I can see is if anyone actually does want to start the
>> description with a dash, in which case they would now need two
>> dashes.  TAP from those producers you mention above would have any
>> dash prefix in the description silently eaten.  Still not ideal, but
>> perhaps not the end of the world.

It's simple enough to deal with, uncommon enough not to be a worry and if you
screw it up it's just cosmetic (that is, it has no effect on passing or
failing).  So I'm not worried.


> IMHO when the TAP grammar changes things that were stable for years
> it's near to the end of the world.
>
> I, for instance, evaluate test results via the description over *lots*
> of archived TAP streams since nearly years. I don't want to become an
> outlaw.

Out of curiosity, what are you using to create these TAP streams and how do
they format the description?


> So I'm strongly with Ovid.
>
> And I don't know why the grammar is about to be discussed at
> all. Until now I thought we would only think about backwards
> compatible extensions that fit into the currently "ignored non-tap"
> design space.
> 
> It would be ok if we were inventing TAP for the first time, but with
> this 20 years old thingie I would be very carefully.

Seeing as how Test::More is the widest producer of TAP, and it's being doing
"ok 1 - description" for years now, and that the dash is optional so nothing
is being made illegal, all of the above would seem to actually be an argument
in favor of adding the optional dash to the grammar as it better describes
reality.

And it produces a nicer to read TAP moving forward, which is important, too.

As a point of reference, we are discussing the grammar exactly because we're
trying to write down reality.  But reality can be hard to nail down when you
try to formalize conventions.  Little niggling things pop up and assumptions
have to be examined.  Does "# SKIP" have to be capitalized or is "# skip" ok,
too? [1]  Do we allow more than one space between tokens? [2]  What's the
character set? [3]  Is "1..0\n" with no skip a passing test? [4]  Is it even
gramatically correct? [5]  What is the TAP end-of-line sequence? [6]  And, of
course, do we enshrine the "ok 1 - description" convention?

All of these little things suddenly become really important when you're trying
to write it all down, in detail, such that others can repeat your work.

To further drill this point into the ground, I noticed a grammatical ambiguity
when generating valid, but nonsense, TAP.  I came across stuff like this:

ok 8lasdjf;akjd;lfj

Which looked like a mistake in the Test-Number grammar until I realized that
was a Test-Result with no Test-Number.  "8lasdjf;akjd;lfj" was the
Description.  This suggests that the following is ambiguous:

ok 8

Is that test 8 passing?  Or is that a test with no number and a description of
"8"?  How do you produce a test with no number and a description of 8?
Without the dash in the grammar, you can't.  You have to say "ok
8not-a-test-number".  You can't even say "ok 8 not a test number" because
that's test number 8 with the description "not a test number".

Un-numbered tests are not just a curiosity (though numeric test descriptions
probably are).  They are important for unsynchronized, parallel testing such
as threads and forking.  The dash allows the test producer to remove this
ambiguity by saying "ok - 8".

Is this important?  Well, TAP is going from being a public interface, to being
a "published" interface.  I mean that in the sense Martin Fowler gave it to
mean "we no longer have control over all implementations and users" and thus
refactoring becomes impossible and backwards compatibility takes on a whole
new meaning.  It also means that people who have no connection to the TAP
community are going to write TAP parsers and producers based purely on the
spec and do REALLY CRAZY SHIT with any crack we leave in the protocol and then
we have to support it FOREVER. [7]

So I assure you that when I bring these things up, I'm trying to nail down the
Jell-O of reality.  I wholeheartedly agree with all my very being that we need
a spec of the current reality of TAP separate from any proposed extensions.

End of unsolicited rant.


[1] Answer, its case insensitive because it's easier to do that in ABNF and
TAP has traditionally been pretty inconsistent about enforcing case.

[2] Still up in the air.  I'm leaning towards more than one space as it gives
more flexibility in formatting for human readability and most parsers (by
which I mean Test::Harness and TAP::Parser) allow more than one space anyway.

[3] Recent discussion seems pretty strong that it's UTF8.

[4] No, though there's no strong reason it couldn't be.

[5] Yes, it is not a parse error.

[6] Newline?  Carriage return + newline?  Both?  This needs to be nailed down
if we're going to pass TAP streams around from machine to machine.

[7] To see this illustrated, please watch rjbs' brilliant rant against mail
formats, "Email Hates The Living".  Also it's funny.
http://video.google.com/videoplay?docid=7054401183589794595


-- 
'All anyone gets in a mirror is themselves,' she said. 'But what you
gets in a good gumbo is everything.'
    -- "Witches Abroad" by Terry Prachett
_______________________________________________
tap mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/tap