Re: [tap] Weird thought for the day: "skip 71 - reason"

Steffen Schwigon <ss5@renormalist.net> Sun, 07 March 2010 11:40 UTC

Return-Path: <ss5@renormalist.net>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C50FD28C15A for <tap@core3.amsl.com>; Sun, 7 Mar 2010 03:40:30 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MWHmtEaLYReL for <tap@core3.amsl.com>; Sun, 7 Mar 2010 03:40:29 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by core3.amsl.com (Postfix) with ESMTP id 5799B28C12E for <tap@ietf.org>; Sun, 7 Mar 2010 03:40:28 -0800 (PST)
Received: by h132974.serverkompetenz.net (Postfix, from userid 1000) id E0BBC9AD4AB; Sun, 7 Mar 2010 12:40:29 +0100 (CET)
To: Ovid <publiustemp-tapx@yahoo.com>
References: <411886.95806.qm@web65707.mail.ac4.yahoo.com>
From: Steffen Schwigon <ss5@renormalist.net>
Date: Sun, 07 Mar 2010 12:40:29 +0100
In-Reply-To: <411886.95806.qm@web65707.mail.ac4.yahoo.com> (Ovid's message of "Fri\, 5 Mar 2010 07\:53\:33 -0800 \(PST\)")
Message-ID: <87ocj0weaa.fsf@renormalist.net>
User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Cc: tap@ietf.org
Subject: Re: [tap] Weird thought for the day: "skip 71 - reason"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 07 Mar 2010 11:40:30 -0000

Ovid <publiustemp-tapx@yahoo.com> writes:
> One thing which bugs me about TAP is this:
>
>   1..5
>   ok 1 - some test
>   ok 2 - another test
>   ok 3 # skip cuz I said so
>   not ok 4 # skip, skip, baby
>   ok 5 -final test
>  
> What does a skipped "not ok" test mean? Its behaviour is
> undefined. This means that different parsers might implement this in
> different ways.  It's an obscure edge case, but with greater
> adoption of TAP, it might crop up.

I do not have a problem with this semtantic gap, I even already used
it, similar to what #TODO does.

That case for instance happens when Test::Builder toolchains are not
available and it is easier to canonically independently generate the
OK/NOTOK part and the directives.


> Here's what I would like to see.
>
>   1..5
>   ok 1 - some test
>   ok 2 - another test
>   skip 3 - skip cuz I said so
>   skip 4 - skip, skip, baby
>   ok 5 -final test
>
>
> In other words, you can't introduce ambiguity here.

“NOT OK # SKIP” is no ambiguity. It does not conflict with anything
else. It is skipped and semantically being “OK”, same as TODO.

The interpretation as SKIP by the TAP::Parser/Aggregator should just
be handled the same emotionless, statistic way as TODO: just count how
many SKIP, and if you like then also how many of them are unexpectedly
OK/NOTOK and left the “semantic strangeness” as exercise to the user.

I *strongly* vote *against* introducing a new value additionally to
OK/NOT_OK. Especially as it doesn't take away the ambiguity of the
“NOT OK # SKIP” case and would be an artificial change introduced
without real urge from users (others than the “Wizards Of TAP”).

I regularly teach TAP to users and the whole point of it's easyness is
the fact I can tell:

  Basically you either report OK or NOT OK (and a plan). 
  That's all. The rest is just sugar.

And I would like to keep the basics and the sugar divided as it
currently is: basics on the left side, and the more sugar the more on
the right.

TAP is KISS: Keep It Simple, Stupid; with emphasis on “keep”.

For non-Perl, non-Test::Builder toolchains that easiness is vital!


> Though this list is for ietf, those who really care about TAP are
> hopefully reading this.

I do care for TAP. 
I use it. 
I teach it.
I build infrastructures around it. 
Perl and non-Perl. In harmony. Quite.

I hope that qualifies my comments a bit. :-)

Kind regards,
Steffen 
-- 
Steffen Schwigon <ss5@renormalist.net>