[tap] Indicating errors in the test suite

Salve J Nilsen <sjn@pvv.org> Fri, 05 March 2010 16:37 UTC

Return-Path: <sjn@pvv.ntnu.no>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost []) by core3.amsl.com (Postfix) with ESMTP id 8A1013A8D4A for <tap@core3.amsl.com>; Fri, 5 Mar 2010 08:37:10 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([]) by localhost (core3.amsl.com []) (amavisd-new, port 10024) with ESMTP id 2W2a203Jra4e for <tap@core3.amsl.com>; Fri, 5 Mar 2010 08:37:09 -0800 (PST)
Received: from decibel.pvv.ntnu.no (decibel.pvv.ntnu.no [IPv6:2001:700:300:1900::1:2]) by core3.amsl.com (Postfix) with ESMTP id A34C93A8965 for <tap@ietf.org>; Fri, 5 Mar 2010 08:37:09 -0800 (PST)
Received: from sjn by decibel.pvv.ntnu.no with local (Exim 4.69) (envelope-from <sjn@pvv.ntnu.no>) id 1NnaWT-0002jM-Js for tap@ietf.org; Fri, 05 Mar 2010 17:37:09 +0100
Date: Fri, 5 Mar 2010 17:37:09 +0100 (CET)
From: Salve J Nilsen <sjn@pvv.org>
X-X-Sender: sjn@decibel.pvv.ntnu.no
To: tap@ietf.org
Message-ID: <alpine.DEB.1.00.1003051725420.2167@decibel.pvv.ntnu.no>
User-Agent: Alpine 1.00 (DEB 882 2007-12-20)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
Subject: [tap] Indicating errors in the test suite
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 05 Mar 2010 16:37:10 -0000


Just wanted to get some clarity, to see if I've understood things 

In any test suite we have basically two error conditions:

1) Errors in the code that's being tested with the test suite. This is
    the main purpose of a test suite.

2) Errors in the test suite code itself.

Detecting cases of 1) is pretty straightforward as long as you have 
tests for it, but for detecting 2) we just have four "indirect" 
indicators of their presence:

1) "not ok # SKIP" - We intended to skip, but failed somehow
2) "ok # TODO" - A test unexpectedly succeeded
3) Mismatch between plan and number of tests that were done.
4) Shell return codes that are unexpected.

- Do we have any other indicators?
- Should we say something about subprocess return codes in the TAP

- Salve

sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <sjn@foo.no>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)