Re: [tap] Weird thought for the day: "skip 71 - reason"

Steffen Schwigon <> Sun, 07 March 2010 11:40 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C50FD28C15A for <>; Sun, 7 Mar 2010 03:40:30 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id MWHmtEaLYReL for <>; Sun, 7 Mar 2010 03:40:29 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id 5799B28C12E for <>; Sun, 7 Mar 2010 03:40:28 -0800 (PST)
Received: by (Postfix, from userid 1000) id E0BBC9AD4AB; Sun, 7 Mar 2010 12:40:29 +0100 (CET)
To: Ovid <>
References: <>
From: Steffen Schwigon <>
Date: Sun, 07 Mar 2010 12:40:29 +0100
In-Reply-To: <> (Ovid's message of "Fri\, 5 Mar 2010 07\:53\:33 -0800 \(PST\)")
Message-ID: <>
User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [tap] Weird thought for the day: "skip 71 - reason"
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 07 Mar 2010 11:40:30 -0000

Ovid <> writes:
> One thing which bugs me about TAP is this:
>   1..5
>   ok 1 - some test
>   ok 2 - another test
>   ok 3 # skip cuz I said so
>   not ok 4 # skip, skip, baby
>   ok 5 -final test
> What does a skipped "not ok" test mean? Its behaviour is
> undefined. This means that different parsers might implement this in
> different ways.  It's an obscure edge case, but with greater
> adoption of TAP, it might crop up.

I do not have a problem with this semtantic gap, I even already used
it, similar to what #TODO does.

That case for instance happens when Test::Builder toolchains are not
available and it is easier to canonically independently generate the
OK/NOTOK part and the directives.

> Here's what I would like to see.
>   1..5
>   ok 1 - some test
>   ok 2 - another test
>   skip 3 - skip cuz I said so
>   skip 4 - skip, skip, baby
>   ok 5 -final test
> In other words, you can't introduce ambiguity here.

“NOT OK # SKIP” is no ambiguity. It does not conflict with anything
else. It is skipped and semantically being “OK”, same as TODO.

The interpretation as SKIP by the TAP::Parser/Aggregator should just
be handled the same emotionless, statistic way as TODO: just count how
many SKIP, and if you like then also how many of them are unexpectedly
OK/NOTOK and left the “semantic strangeness” as exercise to the user.

I *strongly* vote *against* introducing a new value additionally to
OK/NOT_OK. Especially as it doesn't take away the ambiguity of the
“NOT OK # SKIP” case and would be an artificial change introduced
without real urge from users (others than the “Wizards Of TAP”).

I regularly teach TAP to users and the whole point of it's easyness is
the fact I can tell:

  Basically you either report OK or NOT OK (and a plan). 
  That's all. The rest is just sugar.

And I would like to keep the basics and the sugar divided as it
currently is: basics on the left side, and the more sugar the more on
the right.

TAP is KISS: Keep It Simple, Stupid; with emphasis on “keep”.

For non-Perl, non-Test::Builder toolchains that easiness is vital!

> Though this list is for ietf, those who really care about TAP are
> hopefully reading this.

I do care for TAP. 
I use it. 
I teach it.
I build infrastructures around it. 
Perl and non-Perl. In harmony. Quite.

I hope that qualifies my comments a bit. :-)

Kind regards,
Steffen Schwigon <>