Re: [tap] "Nested TAP" in the draft

Salve J Nilsen <> Fri, 05 March 2010 16:12 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 23A0A28C24F for <>; Fri, 5 Mar 2010 08:12:33 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 7u2rz+fvqAKp for <>; Fri, 5 Mar 2010 08:12:29 -0800 (PST)
Received: from ( [IPv6:2001:700:300:1900::1:2]) by (Postfix) with ESMTP id 0850528C2B2 for <>; Fri, 5 Mar 2010 08:12:28 -0800 (PST)
Received: from sjn by with local (Exim 4.69) (envelope-from <>) id 1Nna8X-0004FJ-1p; Fri, 05 Mar 2010 17:12:25 +0100
Date: Fri, 5 Mar 2010 17:12:25 +0100 (CET)
From: Salve J Nilsen <>
To: Ovid <>
In-Reply-To: <>
Message-ID: <>
References: <>
User-Agent: Alpine 1.00 (DEB 882 2007-12-20)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
Cc: TAP <>
Subject: Re: [tap] "Nested TAP" in the draft
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 05 Mar 2010 16:12:33 -0000

Ovid said:
>> From: Salve J Nilsen <>
>> Could you remind us a little of what the purpose of Nested
>> TAP was?
> The main purpose is that it allows multiple TAP streams to be 
> combined into a single stream.  It can be nested to arbitrary 
> depths, but all nesting depths must be consistent.  The first 
> "unindented" line under each nest is the pass/fail status of the 
> nest.  A nest is represented as a single pass/fail result for the 
> outer stream.

[snip explanation]

> One unanswered question regarding nested TAP: are skip and todo 
> allowed on summary test lines?

SKIP means "I expect this test to fail, so I won't run it, but it's 
result should be read as ok anyway." If a set of subtests should be 
summarized with "SKIP", we'll have to know a few things about it:

1) Are ALL the subtests "SKIPped"?

  - In this case I think it's reasonable to summarize with a "ok #
    SKIP" if all subtests were ok, or a "not ok # SKIP" if one of them

2) Are SOME of the subtests SKIPped, while others are regular tests?

  - A pass or a fail in the regular subtests here would throw off the
    summary. If all SKIPped subtest pass, but one regular subtest
    fails, should this be summarized as "not ok # SKIP" or "not ok"?
    Same question goes if both a SKIPped and a regular subtest fails.

  - If all subtests succeed, should it be reported as "ok" or "ok #

  - One way to clean up the summary is to define a "summary priority."
    Which reports are the most important/general?
    (E.g. regular > SKIP > TODO; the most "important" fail ends up in
    the summary)

  - Another way is to require a warning when the subset result is

  - A third way is to extend the protocol to give a more detailed
    report on what the subtest results were. :-P

3) Are NONE of the subtests SKIPped?

  - Summarizing a subset with no SKIPs as SKIPped sounds very much like
    an error to me. :)

I think the case for TODOs are pretty similar, which leaves us with 
the question "How to summarize a mixed result?"

- If all subtests pass, then the summary should be a pass.
- If at least one regular (non-SKIP, non-TODO) subtest fails, then the
   entire subset is summarized with a fail.
- If at least one SKIP subtest fails, but no regular tests, then the
   summary should be a failed SKIP (not ok # SKIP.) -- Treat it as
   one SKIPped test.
- If at least one TODO subtest fails, but no regular tests or SKIPped
   tests fail, then the subset should be reported as passed, with a
   warning ("not ok # TODO Reason".) -- Treat it as one TODO test.
- If there is a mismatch between subtests and summary (e.g. summary is
   given as a failed TODO, but there are failed SKIP tests) then this
   should be a protocol error.

- Salve, throwing in some thoughts to get things going.. :)

sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)