Re: [tap] "Nested TAP" in the draft

Salve J Nilsen <sjn@pvv.org> Fri, 05 March 2010 16:12 UTC

Return-Path: <sjn@pvv.ntnu.no>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 23A0A28C24F for <tap@core3.amsl.com>; Fri, 5 Mar 2010 08:12:33 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7u2rz+fvqAKp for <tap@core3.amsl.com>; Fri, 5 Mar 2010 08:12:29 -0800 (PST)
Received: from decibel.pvv.ntnu.no (decibel.pvv.ntnu.no [IPv6:2001:700:300:1900::1:2]) by core3.amsl.com (Postfix) with ESMTP id 0850528C2B2 for <tap@ietf.org>; Fri, 5 Mar 2010 08:12:28 -0800 (PST)
Received: from sjn by decibel.pvv.ntnu.no with local (Exim 4.69) (envelope-from <sjn@pvv.ntnu.no>) id 1Nna8X-0004FJ-1p; Fri, 05 Mar 2010 17:12:25 +0100
Date: Fri, 5 Mar 2010 17:12:25 +0100 (CET)
From: Salve J Nilsen <sjn@pvv.org>
X-X-Sender: sjn@decibel.pvv.ntnu.no
To: Ovid <publiustemp-tapx@yahoo.com>
In-Reply-To: <91803.89490.qm@web65716.mail.ac4.yahoo.com>
Message-ID: <alpine.DEB.1.00.1003051624280.2167@decibel.pvv.ntnu.no>
References: <91803.89490.qm@web65716.mail.ac4.yahoo.com>
User-Agent: Alpine 1.00 (DEB 882 2007-12-20)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
Cc: TAP <tap@ietf.org>
Subject: Re: [tap] "Nested TAP" in the draft
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 05 Mar 2010 16:12:33 -0000

Ovid said:
>> From: Salve J Nilsen <sjn@pvv.org>
> 
>> Could you remind us a little of what the purpose of Nested
>> TAP was?
>
> The main purpose is that it allows multiple TAP streams to be 
> combined into a single stream.  It can be nested to arbitrary 
> depths, but all nesting depths must be consistent.  The first 
> "unindented" line under each nest is the pass/fail status of the 
> nest.  A nest is represented as a single pass/fail result for the 
> outer stream.

[snip explanation]

> One unanswered question regarding nested TAP: are skip and todo 
> allowed on summary test lines?

SKIP means "I expect this test to fail, so I won't run it, but it's 
result should be read as ok anyway." If a set of subtests should be 
summarized with "SKIP", we'll have to know a few things about it:

1) Are ALL the subtests "SKIPped"?

  - In this case I think it's reasonable to summarize with a "ok #
    SKIP" if all subtests were ok, or a "not ok # SKIP" if one of them
    failed.

2) Are SOME of the subtests SKIPped, while others are regular tests?

  - A pass or a fail in the regular subtests here would throw off the
    summary. If all SKIPped subtest pass, but one regular subtest
    fails, should this be summarized as "not ok # SKIP" or "not ok"?
    Same question goes if both a SKIPped and a regular subtest fails.

  - If all subtests succeed, should it be reported as "ok" or "ok #
    SKIP"?

  - One way to clean up the summary is to define a "summary priority."
    Which reports are the most important/general?
    (E.g. regular > SKIP > TODO; the most "important" fail ends up in
    the summary)

  - Another way is to require a warning when the subset result is
    ambiguous

  - A third way is to extend the protocol to give a more detailed
    report on what the subtest results were. :-P

3) Are NONE of the subtests SKIPped?

  - Summarizing a subset with no SKIPs as SKIPped sounds very much like
    an error to me. :)


I think the case for TODOs are pretty similar, which leaves us with 
the question "How to summarize a mixed result?"

- If all subtests pass, then the summary should be a pass.
- If at least one regular (non-SKIP, non-TODO) subtest fails, then the
   entire subset is summarized with a fail.
- If at least one SKIP subtest fails, but no regular tests, then the
   summary should be a failed SKIP (not ok # SKIP.) -- Treat it as
   one SKIPped test.
- If at least one TODO subtest fails, but no regular tests or SKIPped
   tests fail, then the subset should be reported as passed, with a
   warning ("not ok # TODO Reason".) -- Treat it as one TODO test.
- If there is a mismatch between subtests and summary (e.g. summary is
   given as a failed TODO, but there are failed SKIP tests) then this
   should be a protocol error.


- Salve, throwing in some thoughts to get things going.. :)

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <sjn@foo.no>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)