Re: [tap] Parse error vs failure

Steffen Schwigon <ss5@renormalist.net> Wed, 04 February 2009 22:29 UTC

Return-Path: <ss5@renormalist.net>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0D6263A68F7 for <tap@core3.amsl.com>; Wed, 4 Feb 2009 14:29:40 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.299
X-Spam-Level:
X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_31=0.6]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vGViDzhr6+yb for <tap@core3.amsl.com>; Wed, 4 Feb 2009 14:29:38 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by core3.amsl.com (Postfix) with ESMTP id 903C83A685A for <tap@ietf.org>; Wed, 4 Feb 2009 14:29:38 -0800 (PST)
Received: from ss5 by h132974.serverkompetenz.net with local (Exim 4.63) (envelope-from <ss5@renormalist.net>) id 1LUqFB-0005wZ-HB for tap@ietf.org; Wed, 04 Feb 2009 23:29:17 +0100
To: tap@ietf.org
References: <4984B200.6060907@pobox.com> <439925.21633.qm@web65707.mail.ac4.yahoo.com> <49893C64.1050604@pobox.com>
From: Steffen Schwigon <ss5@renormalist.net>
Date: Wed, 04 Feb 2009 23:29:17 +0100
In-Reply-To: <49893C64.1050604@pobox.com> (Michael G. Schwern's message of "Tue, 03 Feb 2009 22:57:40 -0800")
Message-ID: <87ab91bzuq.fsf@renormalist.net>
User-Agent: Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.19 (linux)
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 22:29:40 -0000

Michael G Schwern <schwern@pobox.com> writes:
> "The color eight died" is a semantic error.  It is nonsense.  Eight
> is not a color and numbers don't die.
>
> You can have something which is syntactically correct but semantic
> nonsense, and there is still meaning in semantic nonsense.

The English comparison is a good idea.

The sentence might still be called lyrics or poetry. No one would try
to disallow this in the English language just for semantic reasons.

I wonder why we cannot model the SKIP semantics similar to #TODO. With
results like "is_actual_ok" to access the raw "ok/not ok" but
interpret the line as "ok" for the stats.


> TAP Version 14
> 1..3
> ok 1
> not ok 2 - no database # SKIP
>   ---
>   file: foo.t
>   line: 42
>   ...
> ok 3
>
> Now, what would you rather tell the user?  If it's just a syntax
> error, then all you get is:
>
>   Syntax error at TAP stream line 4.
>   Diagnostics without test at line 5.

That's why we all already agreed that we do not want it a syntax
error.


> But if you parse and understand it, you can use that information to say:
>
>   Nonsense failing skip test "no database" at foo.t line 42.
>
> pointing the user at the problem.  Isn't that better?

As I said, why not just "fix" the stats, similar to what #TODO does
and provide details via "is_actual_ok" and friends?


Kind regards,
Steffen 
-- 
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>