Re: [tap] "Nested TAP" in the draft

Aristotle Pagaltzis <pagaltzis@gmx.de> Tue, 01 June 2010 06:31 UTC

Return-Path: <pagaltzis@gmx.de>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4FC4128C129 for <tap@core3.amsl.com>; Mon, 31 May 2010 23:31:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.001
X-Spam-Level:
X-Spam-Status: No, score=0.001 tagged_above=-999 required=5 tests=[BAYES_50=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FFonr+MqiFn9 for <tap@core3.amsl.com>; Mon, 31 May 2010 23:31:44 -0700 (PDT)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by core3.amsl.com (Postfix) with SMTP id 8F10C28C144 for <tap@ietf.org>; Mon, 31 May 2010 23:31:43 -0700 (PDT)
Received: (qmail invoked by alias); 01 Jun 2010 06:31:30 -0000
Received: from static-87-79-236-202.netcologne.de (EHLO klangraum) [87.79.236.202] by mail.gmx.net (mp040) with SMTP; 01 Jun 2010 08:31:30 +0200
X-Authenticated: #163624
X-Provags-ID: V01U2FsdGVkX1/Wb61JiUvWd6b2FGJ9HB3GHcwZgJTMSyHLyLAc5U pKzK39uBWOjsjM
Date: Tue, 1 Jun 2010 08:28:06 +0200
From: Aristotle Pagaltzis <pagaltzis@gmx.de>
To: tap@ietf.org
Message-ID: <20100601062806.GB22056@klangraum.plasmasturm.org>
Mail-Followup-To: tap@ietf.org
References: <91803.89490.qm@web65716.mail.ac4.yahoo.com> <alpine.DEB.1.00.1003051624280.2167@decibel.pvv.ntnu.no>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <alpine.DEB.1.00.1003051624280.2167@decibel.pvv.ntnu.no>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Y-GMX-Trusted: 0
Subject: Re: [tap] "Nested TAP" in the draft
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 01 Jun 2010 06:31:45 -0000

* Salve J Nilsen <sjn@pvv.org> [2010-03-05 17:05]:
> Ovid said:
> >One unanswered question regarding nested TAP: are skip and
> >todo allowed on summary test lines?
>
> SKIP means "I expect this test to fail, so I won't run it, but
> it's result should be read as ok anyway." If a set of subtests
> should be summarized with "SKIP", we'll have to know a few
> things about it:
>
> 1) Are ALL the subtests "SKIPped"?
>
>  - In this case I think it's reasonable to summarize with a "ok
>    # SKIP" if all subtests were ok, or a "not ok # SKIP" if one
>    of them failed.
>
> 2) Are SOME of the subtests SKIPped, while others are regular
>    tests?
>
>  - A pass or a fail in the regular subtests here would throw
>    off the summary. If all SKIPped subtest pass, but one
>    regular subtest fails, should this be summarized as "not ok
>    # SKIP" or "not ok"? Same question goes if both a SKIPped
>    and a regular subtest fails.
>
>  - If all subtests succeed, should it be reported as "ok" or
>    "ok # SKIP"?
>
>  - One way to clean up the summary is to define a "summary
>    priority." Which reports are the most important/general?
>    (E.g. regular > SKIP > TODO; the most "important" fail ends
>    up in the summary)
>
>  - Another way is to require a warning when the subset result
>    is ambiguous
>
>  - A third way is to extend the protocol to give a more
>    detailed report on what the subtest results were. :-P
>
> 3) Are NONE of the subtests SKIPped?
>
>  - Summarizing a subset with no SKIPs as SKIPped sounds very
>    much like an error to me. :)
>
>
> I think the case for TODOs are pretty similar, which leaves us
> with the question "How to summarize a mixed result?"
>
> - If all subtests pass, then the summary should be a pass.
> - If at least one regular (non-SKIP, non-TODO) subtest fails,
>   then the entire subset is summarized with a fail.
> - If at least one SKIP subtest fails, but no regular tests,
>   then the summary should be a failed SKIP (not ok # SKIP.) --
>   Treat it as one SKIPped test.
> - If at least one TODO subtest fails, but no regular tests or
>   SKIPped tests fail, then the subset should be reported as
>   passed, with a warning ("not ok # TODO Reason".) -- Treat it
>   as one TODO test.
> - If there is a mismatch between subtests and summary (e.g.
>   summary is given as a failed TODO, but there are failed SKIP
>   tests) then this should be a protocol error.
>
>
> - Salve, throwing in some thoughts to get things going.. :)

Personally this seems to me like too much cleverness.

Don’t forget that the nested TAP is produced on the emitter side.

Keep in mind the two primary (mutually supporting) principles of
TAP:

• That the emitter can be almost completely stupid

• That TAP is descriptive, i.e. merely records results, without
  assigning any particular judgement/interpretation to them

Strictly speaking, even the mere fact that nested TAP requires
the emitter to produce a summary line is a watering down of these
principles: the emitter takes on some aspects of a harness. If we
cannot avoid this, then we should at least minimise the amount of
subtlety involved in emitting this summary line.

The fact that there is a summary line is a backcompat concession
to old harnesses. Personally I would even consider making it some
kind of fixed `not ok # TODO` or such, and *require* harnesses to
look inside the nested sequence if they understand it – so that
emitters could remain completely stupid. Or maybe put this up to
the discretion of the emitter and tell new harnesses never to
rely on the summary. Not sure what exactly would be the best
course of action…

-- 
*AUTOLOAD=*_;sub _{s/(.*)::(.*)/print$2,(",$\/"," ")[defined wantarray]/e;$1}
&Just->another->Perl->hack;
#Aristotle Pagaltzis // <http://plasmasturm.org/>