Re: [tap] Indicating errors in the test suite

Ovid <publiustemp-tapx@yahoo.com> Sat, 06 March 2010 13:36 UTC

Return-Path: <publiustemp-tapx@yahoo.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 84DE23A8F76 for <tap@core3.amsl.com>; Sat, 6 Mar 2010 05:36:30 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MEdWvcVYvFX9 for <tap@core3.amsl.com>; Sat, 6 Mar 2010 05:36:29 -0800 (PST)
Received: from web65714.mail.ac4.yahoo.com (web65714.mail.ac4.yahoo.com [76.13.9.106]) by core3.amsl.com (Postfix) with SMTP id 678AA3A8B66 for <tap@ietf.org>; Sat, 6 Mar 2010 05:36:29 -0800 (PST)
Received: (qmail 32397 invoked by uid 60001); 6 Mar 2010 13:36:29 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1267882589; bh=S6DzKxYXlvv9LE3FwT5Xbvyx5faBDXqUA0lJSHTrsEQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=AiPTwikI2L8iJPUTz2jPy1+P897v8NpU3qsPqtL6kpypxOiJKq+5ZAauRc27zagVakMjOW8ozU1jjv9EISF3p0Trnx4Vx4wdiQZegYDgOsr2xkmznfUjPwAzuaQRHqIbtyNnXrnZeMQkZdbSSZlqyyNwTIrjpXGtxYOQcJX1lU8=
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=2ePyCtrN8jigE5cA/8Uf+geXHaHHPNVOXUSYnWHwvfjkIkiBVnqrOy8gVrbCpXdEAz+yt2MSMCP7r8MA2vqj6BQsSc1O6LhWW25HGXbaZeK53eg6MA1xnP3hZWdgNAVonXwq04zrwhG4svp9Ae/f3hgl4aQ2KXh0R5RuynFT9cE=;
Message-ID: <803630.31877.qm@web65714.mail.ac4.yahoo.com>
X-YMail-OSG: 2FweeHsVM1lGiJr9vHjbtsFzclTyKcOh12C4u2ZoEbqdfZTAo1F1DyNimX_bJE1psLD2Z_KoyBmQmyKBYeOfCpW7jnT3QbCWQ.o0dGhtxoEim2oILXPWkId9qSK2yVyk2S8awIvuSYN93XdqgJMHC8cju1o6sN_RYaoTISwbDUpAS.lVIuuLZ3dpPPQBuOtucqOxImsS_NA1qqPDfbxa1GbCUk.SPrySYbcy6QYNQZ8kUnIC_kwX4k4m_Uim7wgUJBm87MwxuRn.y4EbEB1oNTmpyhD6yjzMCEv5e.6dOX2Icv5dA05II_d5vHdNWOTiej8_cp6Zfaw1i5G5piR4Xacn.zAQiQ--
Received: from [90.221.155.100] by web65714.mail.ac4.yahoo.com via HTTP; Sat, 06 Mar 2010 05:36:29 PST
X-Mailer: YahooMailRC/300.3 YahooMailWebService/0.8.100.260964
References: <alpine.DEB.1.00.1003051725420.2167@decibel.pvv.ntnu.no>
Date: Sat, 6 Mar 2010 05:36:29 -0800 (PST)
From: Ovid <publiustemp-tapx@yahoo.com>
To: Salve J Nilsen <sjn@pvv.org>, tap@ietf.org
In-Reply-To: <alpine.DEB.1.00.1003051725420.2167@decibel.pvv.ntnu.no>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [tap] Indicating errors in the test suite
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 06 Mar 2010 13:36:30 -0000

----- Original Message ----
> From: Salve J Nilsen <sjn@pvv.org>


> 1) "not ok # SKIP" - We intended to skip, but failed somehow
> 2) "ok # TODO" - A test unexpectedly succeeded
> 3) Mismatch between plan and number of tests that were done.
> 4) Shell return codes that are unexpected.
> 
> - Do we have any other indicators?
> - Should we say something about subprocess return codes in the TAP
>   standard?

Returns codes are unreliable. You don't get them if you're running TAP in Javascript.  If some intermediate process is handing the TAP to the consumer and that intermediate process fails but the code you were testing did not, relying on the return code will give you a false negative.

Cheers,
Ovid --
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog - http://use.perl.org/~Ovid/journal/
Twitter - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6