Re: [tap] Indicating errors in the test suite

Salve J Nilsen <sjn@pvv.org> Sat, 06 March 2010 14:40 UTC

Return-Path: <sjn@pvv.ntnu.no>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1EA133A8B93 for <tap@core3.amsl.com>; Sat, 6 Mar 2010 06:40:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 79UvqudgEvH6 for <tap@core3.amsl.com>; Sat, 6 Mar 2010 06:40:03 -0800 (PST)
Received: from decibel.pvv.ntnu.no (decibel.pvv.ntnu.no [IPv6:2001:700:300:1900::1:2]) by core3.amsl.com (Postfix) with ESMTP id 159333A88E3 for <tap@ietf.org>; Sat, 6 Mar 2010 06:40:02 -0800 (PST)
Received: from sjn by decibel.pvv.ntnu.no with local (Exim 4.69) (envelope-from <sjn@pvv.ntnu.no>) id 1NnvAf-0000at-Jm; Sat, 06 Mar 2010 15:40:01 +0100
Date: Sat, 6 Mar 2010 15:40:01 +0100 (CET)
From: Salve J Nilsen <sjn@pvv.org>
X-X-Sender: sjn@decibel.pvv.ntnu.no
To: Ovid <publiustemp-tapx@yahoo.com>
In-Reply-To: <803630.31877.qm@web65714.mail.ac4.yahoo.com>
Message-ID: <alpine.DEB.1.00.1003061530060.2167@decibel.pvv.ntnu.no>
References: <alpine.DEB.1.00.1003051725420.2167@decibel.pvv.ntnu.no> <803630.31877.qm@web65714.mail.ac4.yahoo.com>
User-Agent: Alpine 1.00 (DEB 882 2007-12-20)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: tap@ietf.org
Subject: Re: [tap] Indicating errors in the test suite
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 06 Mar 2010 14:40:04 -0000

Ovid said:
>> From: Salve J Nilsen <sjn@pvv.org>
>
>> 1) "not ok # SKIP" - We intended to skip, but failed somehow
>> 2) "ok # TODO" - A test unexpectedly succeeded
>> 3) Mismatch between plan and number of tests that were done.
>> 4) Shell return codes that are unexpected.
>> 
>> - Do we have any other indicators?
>> - Should we say something about subprocess return codes in the TAP
>>   standard?
>
> Returns codes are unreliable. You don't get them if you're running 
> TAP in Javascript.  If some intermediate process is handing the TAP 
> to the consumer and that intermediate process fails but the code you 
> were testing did not, relying on the return code will give you a 
> false negative.

Would it be sensibel to just say "We expect a non-error return code if 
it's available" and leave it at that? (Basically saying nothing about 
implementation details of the specific environment, but leaving it to 
the producers and consumers to give a warning if an unexpected return 
code shows up somewhere.)

My point is that even if return codes are unreliable they can still be 
useful indicators of something funky going on somewhere, and because 
of this, the TAP document should at least mention it.

E.g. "TAP producers and consumers should emit a warning if any 
non-successful process return code is detected."

Or to say it in a different way, "don't throw away return codes that 
might mean something, even if they're unreliable."


- Salve

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <sjn@foo.no>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)