Re: [tap] RFC Status?

["Bruno P. Kinoshita" <[e-mail]>]

Hi Ovid!

> Another thing to note about TAP is that we still don't have structured diagnostics.

Indeed. Most projects in Jenkins, even today, rely on JUnit for reporting test results. They have a somewhat modified XML schema for handling attachments in JUnit (like in the junit-attachments-plugin). 

In tap4j, tap-plugin and testlink-plugin, we used the YAML blocks in our TAP stream.

ok 1
  ---
  name: test IO
  description: yadda yadda
  ...
not ok 2
1..2

> I argued that was madness and having a single standard would make parsing much easier.

+1 for some bugs in tap4j, I was very glad I could point users to the TAP 13 details page, and tell them that we would have to stick with the docs in testanything.org. That still helps a lot.

> Putting that aside, there were also arguments about how and what information should be captured in said diagnostics. Filename and line  number are clear, but what about tests for which that information doesn't make sense? Or what about a timestamp? Local time? GMT (useless)? UTC?

Agreed too, we adopted the main entries from [1] in tap4j, plus the extensions entry (which is free to have whatever users want).

While this works well for tap4j and the jenkins plug-in, a standardization would definitely help, specially to integrate with other tools.

> Ask any questions you like.

What do you think that would be the next steps to push forward a TAP 14 draft, and make some progress on the RFC? Maybe something like monthly meetings in IRC or some other IM, and some discussion panels in hackathons/meetups? 

Thanks!
Bruno

[1] https://web.archive.org/web/20111017073902/http://testanything.org/wiki/index.php/TAP_diagnostic_syntax


>________________________________
> From: Ovid <[e-mail]>
>To: Andrew de Andrade <[e-mail]>; Bruno P. Kinoshita <[e-mail]> 
>Cc: "[e-mail]" <[e-mail]> 
>Sent: Tuesday, August 12, 2014 4:49 AM
>Subject: Re: [tap] RFC Status?
> 
>
>
>Should the aggregate reporter report subtests as they are received or should it keep the subtest results that finish out of order in a buffer and always report the tests in a deterministic way? I did see this in the readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that it is left up to the TAP aggregator consuming the stream to either report things in order with or without numbers (if desired) or out of order, but un-numbered. 
>The following assumes that I correctly understood your question :)
>
>
>Presumably an aggregate reporter either runs tests in order (in which case tests come back in order), or or runs them in parallel (in which case order is not guaranteed). Let's say that your test harness runs test programs A and B. It cannot control their output, so A and B must guarantee that they return valid TAP. What that means in the case of subtests is the following.
>
>
>
>Let's say that A returns:
>1..2
>>ok 1 - We have lift off!
>>not ok 2 - Houston, we have a problem
>And B returns this:
>ok 1 - Trap is set
>>ok 2 - Trap is sprung
>>ok 3 - Compromising photos of boss emailed
>>1..3
>With that, if we run in parallel, either of those TAP chunks can be returned first, but until the consumer receives the entire output from A or B (in whatever order), it can produce no output. It could, in theory, block on one, but this behavior isn't defined because the specification is about the TAP producers, not the consumers. Maybe that that should be rectified? (Or maybe it is also about the consumers and it's been too long since I worked on it and just plum forgot).
>
>
>Assuming all goes well, the consumer can produce:
>    1..2
>>    ok 1 - We have lift off!
>>    not ok 2 - Houston, we have a problem
>>not ok 1 - A
>>    ok 1 - Trap is set
>>    ok 2 - Trap is sprung
>>    ok 3 - Compromising photos of boss emailed
>>    1..3
>>ok 2 - B
>>1..2
>Or this:
>    ok 1 - Trap is set
>>    ok 2 - Trap is sprung
>>    ok 3 - Compromising photos of boss emailed
>>    1..3
>>ok 1 - B
>>    1..2
>>    ok 1 - We have lift off!
>>    not ok 2 - Houston, we have a problem
>>not ok 2 - A
>>1..2
>Or (but not recommended):
>not ok 1 - A
>>ok 2 - B
>>1..2
>The latter is not recommended because you're throwing away information, but it should parse the same way as the previous ones with an older TAP parser because ok/not ok binds to the beginning of the line (and thus indented TAP is ignored).
>
>
>In fact, you could put the 1..2 plans before the tests if you wanted to (also called a "leading plan").
>
>
>The reason the nested TAP behavior isn't well-specified is because there were many arguments over it at conferences and hackathons, and then one day (I was in Norway, I think), I got tired of the arguments and just sat down and implemented a mostly backwards-compatible solution. Implementation trumps opinion, so I won.
>
>
>Another thing to note about TAP is that we still don't have structured diagnostics. Again, this was largely because no one could agree on what they should look like. Some wanted producers to be able to output them in whatever format they wanted (JSON, YAML, XML, and so on). I argued that was madness and having a single standard would make parsing much easier. Putting that aside, there were also arguments about how and what information should be captured in said diagnostics. Filename and line number are clear, but what about tests for which that information doesn't make sense? Or what about a timestamp? Local time? GMT (useless)? UTC?
>
>
>So there's lots of work to be done and lots of behavior to be nailed down. Fortunately, TAP's been going strong since the 1980s, but it's only been the past decade or so since it's been named and formally developed.
>
>
>Ask any questions you like.
>
>
>Best,
>Ovid
>--
>IT consulting, training, international recruiting
>       http://www.allaroundtheworld.fr/.
>Buy my book! - http://bit.ly/beginning_perl
>Live and work overseas - http://www.overseas-exile.com/
>
>
>
>
>
>
>On Monday, 11 August 2014, 23:31, Andrew de Andrade <[e-mail]> wrote:
> 
>
>>
>>
>>Yes, this all helps a bunch. I've been using substack's tape (https://github.com/substack/tape) and isaacs' tap (https://github.com/isaacs/node-tap) modules thus far. 
>>
>>
>>tape currently supports v13, which as you pointed out Bruno, does not formally specify how nesting should work, and it does support defining tests in a nested way, but does not report them as nested. Instead, it simply updates the plan count and continues with everything flat/unindented and reports the plan count at the end. For example this code...
>>
>>
>>```
>>var test = require('tape');
>>
>>
>>test('foobar', function(t) {
>>  t.plan(3);
>>  //var test = tape.createHarness();
>>
>>
>>  t.ok(true, 'foo');
>>  t.ok(true, 'bar');
>>
>>
>>  t.test('fibble', function(tt) {
>>    tt.plan(2);
>>    tt.ok(true, 'wobble');
>>    tt.ok(true, 'wizzle');
>>  });
>>});
>>```
>>...produces...
>>
>>
>>```
>>TAP version 13
>># foobar
>>ok 1 foo
>>ok 2 bar
>># fibble
>>ok 3 wobble
>>ok 4 wizzle
>>
>>
>>1..4
>># tests 4
>># pass  4
>>
>>
>># ok
>>```
>>
>>
>>What I did just find is an implementation from Ovid from earlier this year in March:
>>
>>
>>http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
>>
>>https://github.com/Ovid/tap-stream
>>
>>
>>
>>This stream approach is really nice, but the only thing that leaves me scratching my head is that is implies that asynchrony is permitted (parallelized tests, aggregation, out-of-order reporting), but I'm not exactly sure what the implications for this are for the aggregate tap producer. 
>>
>>
>>Should the aggregate reporter report subtests as they are received or should it keep the subtest results that finish out of order in a buffer and always report the tests in a deterministic way? I did see this in the readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that it is left up to the TAP aggregator consuming the stream to either report things in order with or without numbers (if desired) or out of order, but un-numbered. 
>>
>>
>>Given my intended use case, buffering results so that results can always be reported numbered in order would be desirable, since I want to be able to diff TAP results across test runs under different conditions (e.g. different dependency versions). Has anyone done any work with diffing TAP results?
>>
>>Bruno, is tap4j purely synchronous or at least in order or can subtests be asynchronous and reported out of order?
>>
>>
>>Seeing that Ovid is still actively working on this stuff (comments on tap on perlqa3 as recent as April this year), is it save to assume that there are ideas here that will make it into version 14? The test blocks you pointed to does reference version 14 in the example -->  https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks
>>
>>
>>
>>_______________________________________________
>>tap mailing list
>>[e-mail]
>>https://www.ietf.org/mailman/listinfo/tap
>>
>>
>>
>
>