[tap] Descriptive, not prescriptive [was: Indicating errors in the test suite]

Aristotle Pagaltzis <pagaltzis@gmx.de> Mon, 15 March 2010 12:46 UTC

Return-Path: <pagaltzis@gmx.de>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 18CD23A6C2C for <tap@core3.amsl.com>; Mon, 15 Mar 2010 05:46:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.001
X-Spam-Level:
X-Spam-Status: No, score=0.001 tagged_above=-999 required=5 tests=[BAYES_50=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pvMqoL0x6OfW for <tap@core3.amsl.com>; Mon, 15 Mar 2010 05:46:33 -0700 (PDT)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by core3.amsl.com (Postfix) with SMTP id 4D5EF3A6CB0 for <tap@ietf.org>; Mon, 15 Mar 2010 05:26:46 -0700 (PDT)
Received: (qmail invoked by alias); 15 Mar 2010 12:26:52 -0000
Received: from static-87-79-236-202.netcologne.de (EHLO klangraum) [87.79.236.202] by mail.gmx.net (mp063) with SMTP; 15 Mar 2010 13:26:52 +0100
X-Authenticated: #163624
X-Provags-ID: V01U2FsdGVkX188crhK2+08kjkbcAD63OMLmqQNtJPh6pHMhHl+Fn 2wAIzRRLnhyZyF
Date: Mon, 15 Mar 2010 13:26:29 +0100
From: Aristotle Pagaltzis <pagaltzis@gmx.de>
To: tap@ietf.org
Message-ID: <20100315122629.GC27881@klangraum.plasmasturm.org>
Mail-Followup-To: tap@ietf.org
References: <alpine.DEB.1.00.1003051725420.2167@decibel.pvv.ntnu.no> <803630.31877.qm@web65714.mail.ac4.yahoo.com> <alpine.DEB.1.00.1003061530060.2167@decibel.pvv.ntnu.no>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <alpine.DEB.1.00.1003061530060.2167@decibel.pvv.ntnu.no>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Y-GMX-Trusted: 0
X-FuHaFi: 0.64000000000000001
Subject: [tap] Descriptive, not prescriptive [was: Indicating errors in the test suite]
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Mar 2010 12:46:34 -0000

* Salve J Nilsen <sjn@pvv.org> [2010-03-06 15:35]:
> Or to say it in a different way, "don't throw away return codes
> that might mean something, even if they're unreliable."

I agree.

We already have a similar issue in other cases, eg. `ok # TODO`:
sometimes you want to treat those as PASS and sometimes as FAIL,
and the test itself cannot know which interpretation is right,
and the harness, only at that point in time. If you’re archiving
TAP documents, you might use a different interpretation of the
same one test result at different points in time depending on
what you are mining your data for.

So TAP should not come with value judgement included. It should
merely capture outcomes. It is up to the consumer at the point in
time where it is parsing the TAP to make these judgements.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>