Re: [tap] "Nested TAP" in the draft

[Aristotle Pagaltzis <[e-mail]>]

* Salve J Nilsen <[e-mail]> [2010-03-05 17:05]:
> Ovid said:
> >One unanswered question regarding nested TAP: are skip and
> >todo allowed on summary test lines?
>
> SKIP means "I expect this test to fail, so I won't run it, but
> it's result should be read as ok anyway." If a set of subtests
> should be summarized with "SKIP", we'll have to know a few
> things about it:
>
> 1) Are ALL the subtests "SKIPped"?
>
>  - In this case I think it's reasonable to summarize with a "ok
>    # SKIP" if all subtests were ok, or a "not ok # SKIP" if one
>    of them failed.
>
> 2) Are SOME of the subtests SKIPped, while others are regular
>    tests?
>
>  - A pass or a fail in the regular subtests here would throw
>    off the summary. If all SKIPped subtest pass, but one
>    regular subtest fails, should this be summarized as "not ok
>    # SKIP" or "not ok"? Same question goes if both a SKIPped
>    and a regular subtest fails.
>
>  - If all subtests succeed, should it be reported as "ok" or
>    "ok # SKIP"?
>
>  - One way to clean up the summary is to define a "summary
>    priority." Which reports are the most important/general?
>    (E.g. regular > SKIP > TODO; the most "important" fail ends
>    up in the summary)
>
>  - Another way is to require a warning when the subset result
>    is ambiguous
>
>  - A third way is to extend the protocol to give a more
>    detailed report on what the subtest results were. :-P
>
> 3) Are NONE of the subtests SKIPped?
>
>  - Summarizing a subset with no SKIPs as SKIPped sounds very
>    much like an error to me. :)
>
>
> I think the case for TODOs are pretty similar, which leaves us
> with the question "How to summarize a mixed result?"
>
> - If all subtests pass, then the summary should be a pass.
> - If at least one regular (non-SKIP, non-TODO) subtest fails,
>   then the entire subset is summarized with a fail.
> - If at least one SKIP subtest fails, but no regular tests,
>   then the summary should be a failed SKIP (not ok # SKIP.) --
>   Treat it as one SKIPped test.
> - If at least one TODO subtest fails, but no regular tests or
>   SKIPped tests fail, then the subset should be reported as
>   passed, with a warning ("not ok # TODO Reason".) -- Treat it
>   as one TODO test.
> - If there is a mismatch between subtests and summary (e.g.
>   summary is given as a failed TODO, but there are failed SKIP
>   tests) then this should be a protocol error.
>
>
> - Salve, throwing in some thoughts to get things going.. :)

Personally this seems to me like too much cleverness.

Don’t forget that the nested TAP is produced on the emitter side.

Keep in mind the two primary (mutually supporting) principles of
TAP:

• That the emitter can be almost completely stupid

• That TAP is descriptive, i.e. merely records results, without
  assigning any particular judgement/interpretation to them

Strictly speaking, even the mere fact that nested TAP requires
the emitter to produce a summary line is a watering down of these
principles: the emitter takes on some aspects of a harness. If we
cannot avoid this, then we should at least minimise the amount of
subtlety involved in emitting this summary line.

The fact that there is a summary line is a backcompat concession
to old harnesses. Personally I would even consider making it some
kind of fixed `not ok # TODO` or such, and *require* harnesses to
look inside the nested sequence if they understand it – so that
emitters could remain completely stupid. Or maybe put this up to
the discretion of the emitter and tell new harnesses never to
rely on the summary. Not sure what exactly would be the best
course of action…

-- 
*AUTOLOAD=*_;sub _{s/(.*)::(.*)/print$2,(",$\/"," ")[defined wantarray]/e;$1}
&Just->another->Perl->hack;
#Aristotle Pagaltzis // <http://plasmasturm.org/>