Re: [tap] "Nested TAP" in the draft

[Salve J Nilsen <[e-mail]>]

Ovid said:
>> From: Salve J Nilsen <[e-mail]>
> 
>> Could you remind us a little of what the purpose of Nested
>> TAP was?
>
> The main purpose is that it allows multiple TAP streams to be 
> combined into a single stream.  It can be nested to arbitrary 
> depths, but all nesting depths must be consistent.  The first 
> "unindented" line under each nest is the pass/fail status of the 
> nest.  A nest is represented as a single pass/fail result for the 
> outer stream.

[snip explanation]

> One unanswered question regarding nested TAP: are skip and todo 
> allowed on summary test lines?

SKIP means "I expect this test to fail, so I won't run it, but it's 
result should be read as ok anyway." If a set of subtests should be 
summarized with "SKIP", we'll have to know a few things about it:

1) Are ALL the subtests "SKIPped"?

  - In this case I think it's reasonable to summarize with a "ok #
    SKIP" if all subtests were ok, or a "not ok # SKIP" if one of them
    failed.

2) Are SOME of the subtests SKIPped, while others are regular tests?

  - A pass or a fail in the regular subtests here would throw off the
    summary. If all SKIPped subtest pass, but one regular subtest
    fails, should this be summarized as "not ok # SKIP" or "not ok"?
    Same question goes if both a SKIPped and a regular subtest fails.

  - If all subtests succeed, should it be reported as "ok" or "ok #
    SKIP"?

  - One way to clean up the summary is to define a "summary priority."
    Which reports are the most important/general?
    (E.g. regular > SKIP > TODO; the most "important" fail ends up in
    the summary)

  - Another way is to require a warning when the subset result is
    ambiguous

  - A third way is to extend the protocol to give a more detailed
    report on what the subtest results were. :-P

3) Are NONE of the subtests SKIPped?

  - Summarizing a subset with no SKIPs as SKIPped sounds very much like
    an error to me. :)


I think the case for TODOs are pretty similar, which leaves us with 
the question "How to summarize a mixed result?"

- If all subtests pass, then the summary should be a pass.
- If at least one regular (non-SKIP, non-TODO) subtest fails, then the
   entire subset is summarized with a fail.
- If at least one SKIP subtest fails, but no regular tests, then the
   summary should be a failed SKIP (not ok # SKIP.) -- Treat it as
   one SKIPped test.
- If at least one TODO subtest fails, but no regular tests or SKIPped
   tests fail, then the subset should be reported as passed, with a
   warning ("not ok # TODO Reason".) -- Treat it as one TODO test.
- If there is a mismatch between subtests and summary (e.g. summary is
   given as a failed TODO, but there are failed SKIP tests) then this
   should be a protocol error.


- Salve, throwing in some thoughts to get things going.. :)

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <[e-mail]>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)