Re: [tap] RFC Status?

"Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br> Tue, 12 August 2014 14:22 UTC

Return-Path: <brunodepaulak@yahoo.com.br>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 160D41A0869 for <tap@ietfa.amsl.com>; Tue, 12 Aug 2014 07:22:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.75
X-Spam-Level:
X-Spam-Status: No, score=-0.75 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_31=0.6, RCVD_IN_DNSWL_NONE=-0.0001, REPTO_QUOTE_YAHOO=0.646, URIBL_DBL_ABUSE_REDIR=0.001, URIBL_DBL_REDIR=0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id md2aiOk_MZi6 for <tap@ietfa.amsl.com>; Tue, 12 Aug 2014 07:21:57 -0700 (PDT)
Received: from nm19-vm6.bullet.mail.gq1.yahoo.com (nm19-vm6.bullet.mail.gq1.yahoo.com [98.136.217.29]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 72FD11A06D4 for <tap@ietf.org>; Tue, 12 Aug 2014 07:21:57 -0700 (PDT)
Received: from [216.39.60.182] by nm19.bullet.mail.gq1.yahoo.com with NNFMP; 12 Aug 2014 14:21:57 -0000
Received: from [98.137.12.212] by tm18.bullet.mail.gq1.yahoo.com with NNFMP; 12 Aug 2014 14:21:57 -0000
Received: from [127.0.0.1] by omp1020.mail.gq1.yahoo.com with NNFMP; 12 Aug 2014 14:21:57 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 137501.57638.bm@omp1020.mail.gq1.yahoo.com
Received: (qmail 18910 invoked by uid 60001); 12 Aug 2014 14:21:57 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.br; s=s1024; t=1407853317; bh=JPGSmj9e9Ycp93EPTmANdvTRbqHiuN/GoBzCOwnEYqk=; h=References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=Z00W78lVzclA5QE9lD3m4Ti9M0Hzu+hfyuiFiEqtAdNth5ooQyf/XxtU7vOopiF7S6mxBnY9v23HzJoCPJe3zHksHbw6toPeWVwWHOLsbxp5ZoDj71AzyWM/l0HJPm5Pmj1/1c+8P93+7ooVFnTw1ZgD8IUlUrPd/g9QgX1jG6s=
X-YMail-OSG: 55RMz_EVM1l7A5qrC.k7uLDT0yj59s69sca_wBZ5ieEluQG RQ_b6ozCAppRreXWENpUzjcLr.8Dg.IXiEHN08o5ZW7CxFViptdHkdS_Ncen qfeEDV10KOwOr.NcZNxO7UbbMNuzl1Z8ni11w.5wIRpFVzD_mma_TWC3yJI. N6mYkpM6OCEv1wYu.JVGKShTs1VDkuPWuZUDHiqwWZxIaZL9kI6LMT5fxXPd nJAxkAMgMG2PKCoF7NGW604v.PIru3Yio9bg3S6Ll9Q_zOhPCIUQZTYze6DF auyn.Q2c0v.qb1Ueg5OGgS.49HBsm332Org_ERHiR.XxXtO_4EATid.6ENOM BmjCKSK9tz.64hA6OAOG.7A_FmZ_cuCM0TuYpiNLzoVBmlOj8iAUIWmJBCTc .QsdZzNhT1MZ6eEZA1L6nAD6pf4GT.ePSbJOKQs.N8sxpyIy68VsxFERg3uy 4zgQ6JpoSw3ANIqkl_5PyK6lK2JdrpBdak9y.NuCXrKYwN5ACq.HGFuWbmzV weUXiU4hci_B1FmtuAEXcgl.rxXc4KlA4LzxTnEuyOecfJ36JJx3mufOgLud XI4ds7SBlwkS55VIfaISAQHhpisPQuEanlxTsSeaJ6V.yiUNQXL7gQtSukrl x_lTVFQrkLlroli_jl0s_ugAxq9SEpqr0kl_EpM5I4dc5xYAVO2Xk34PA78U BqR_t5Cg8kWPU8yMJhxYFWNAyCTc_csUwuEc1wYnnGwlN.IV3k.5qjR_L6bO eCZkxvD8NVFyzSE9zqJNPHZkLcCRbZQRd0LoCPdjodT20loSC8ID1zuGJdY4 utePbQ_mHMjUEzeXZS4fub_eAEsjjSprXbZc.1WG0W7EYHDhSM3riNMuz0jF z8fV31rbQRk8PewR9CtHTxVqUeBMMrWKBGzO1oFtBUlQPcQm8noEb515__zj l9xj78aKmuY7GA8o8mriPee8ugywLJVPc2mCGmF_r_7js.BTCNXWU9MOGuHQ AvNN8qpOwy74fXerlzDoMj.Ob7F4fPTD8Y.JS4Iam9H3wnBVMTyq20v.hq1w 2t5dR_C5fp2otSECHgjsTTLs0dBbDWfo_lwG_FZG76gUBRzr148BicQ8D9CV hkMdh8J9my2W3bFOFqXdTC219t2bfg_vqUvtJULLMQOPrQYXD6TvxCKKOqYX bI9ib42qTpanMiBDEim4fs8x55_vG10BrAHQShkE-
Received: from [189.120.175.52] by web163504.mail.gq1.yahoo.com via HTTP; Tue, 12 Aug 2014 07:21:56 PDT
X-Rocket-MIMEInfo: 002.001, SGkgT3ZpZCEKCj7CoEFub3RoZXIgdGhpbmcgdG8gbm90ZSBhYm91dCBUQVAgaXMgdGhhdCB3ZSBzdGlsbCBkb24ndCBoYXZlIHN0cnVjdHVyZWQgZGlhZ25vc3RpY3MuCgpJbmRlZWQuIE1vc3QgcHJvamVjdHMgaW4gSmVua2lucywgZXZlbiB0b2RheSwgcmVseSBvbiBKVW5pdCBmb3IgcmVwb3J0aW5nIHRlc3QgcmVzdWx0cy4gVGhleSBoYXZlIGEgc29tZXdoYXQgbW9kaWZpZWQgWE1MIHNjaGVtYSBmb3IgaGFuZGxpbmcgYXR0YWNobWVudHMgaW4gSlVuaXQgKGxpa2UgaW4gdGhlIGp1bml0LWF0dGFjaG1lbnQBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.201.700
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <7250AD76-F5DF-494C-8A00-B4320053EBE7@ggvaidya.com> <1407788544.42539.YahooMailNeo@web163506.mail.gq1.yahoo.com> <CAP4gcswppT=HEy4ew-TejcvJjWT8AwXXRjnG2ECZa7tWnbT7ZQ@mail.gmail.com> <1407829749.48636.YahooMailNeo@web126105.mail.ne1.yahoo.com>
Message-ID: <1407853316.14571.YahooMailNeo@web163504.mail.gq1.yahoo.com>
Date: Tue, 12 Aug 2014 07:21:56 -0700
From: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
To: Ovid <curtis_ovid_poe@yahoo.com>, Andrew de Andrade <andrew@deandrade.com.br>
In-Reply-To: <1407829749.48636.YahooMailNeo@web126105.mail.ne1.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="1159194385-1655390233-1407853316=:14571"
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/Hy11B3hcjP_bC1pUXZXFv1-XIqA
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Aug 2014 14:22:03 -0000

Hi Ovid!

> Another thing to note about TAP is that we still don't have structured diagnostics.

Indeed. Most projects in Jenkins, even today, rely on JUnit for reporting test results. They have a somewhat modified XML schema for handling attachments in JUnit (like in the junit-attachments-plugin). 

In tap4j, tap-plugin and testlink-plugin, we used the YAML blocks in our TAP stream.

ok 1
  ---
  name: test IO
  description: yadda yadda
  ...
not ok 2
1..2

> I argued that was madness and having a single standard would make parsing much easier.

+1 for some bugs in tap4j, I was very glad I could point users to the TAP 13 details page, and tell them that we would have to stick with the docs in testanything.org. That still helps a lot.

> Putting that aside, there were also arguments about how and what information should be captured in said diagnostics. Filename and line  number are clear, but what about tests for which that information doesn't make sense? Or what about a timestamp? Local time? GMT (useless)? UTC?

Agreed too, we adopted the main entries from [1] in tap4j, plus the extensions entry (which is free to have whatever users want).

While this works well for tap4j and the jenkins plug-in, a standardization would definitely help, specially to integrate with other tools.

> Ask any questions you like.

What do you think that would be the next steps to push forward a TAP 14 draft, and make some progress on the RFC? Maybe something like monthly meetings in IRC or some other IM, and some discussion panels in hackathons/meetups? 

Thanks!
Bruno

[1] https://web.archive.org/web/20111017073902/http://testanything.org/wiki/index.php/TAP_diagnostic_syntax


>________________________________
> From: Ovid <curtis_ovid_poe@yahoo.com>
>To: Andrew de Andrade <andrew@deandrade.com.br>br>; Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> 
>Cc: "tap@ietf.org" <tap@ietf.org> 
>Sent: Tuesday, August 12, 2014 4:49 AM
>Subject: Re: [tap] RFC Status?
> 
>
>
>Should the aggregate reporter report subtests as they are received or should it keep the subtest results that finish out of order in a buffer and always report the tests in a deterministic way? I did see this in the readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that it is left up to the TAP aggregator consuming the stream to either report things in order with or without numbers (if desired) or out of order, but un-numbered. 
>The following assumes that I correctly understood your question :)
>
>
>Presumably an aggregate reporter either runs tests in order (in which case tests come back in order), or or runs them in parallel (in which case order is not guaranteed). Let's say that your test harness runs test programs A and B. It cannot control their output, so A and B must guarantee that they return valid TAP. What that means in the case of subtests is the following.
>
>
>
>Let's say that A returns:
>1..2
>>ok 1 - We have lift off!
>>not ok 2 - Houston, we have a problem
>And B returns this:
>ok 1 - Trap is set
>>ok 2 - Trap is sprung
>>ok 3 - Compromising photos of boss emailed
>>1..3
>With that, if we run in parallel, either of those TAP chunks can be returned first, but until the consumer receives the entire output from A or B (in whatever order), it can produce no output. It could, in theory, block on one, but this behavior isn't defined because the specification is about the TAP producers, not the consumers. Maybe that that should be rectified? (Or maybe it is also about the consumers and it's been too long since I worked on it and just plum forgot).
>
>
>Assuming all goes well, the consumer can produce:
>    1..2
>>    ok 1 - We have lift off!
>>    not ok 2 - Houston, we have a problem
>>not ok 1 - A
>>    ok 1 - Trap is set
>>    ok 2 - Trap is sprung
>>    ok 3 - Compromising photos of boss emailed
>>    1..3
>>ok 2 - B
>>1..2
>Or this:
>    ok 1 - Trap is set
>>    ok 2 - Trap is sprung
>>    ok 3 - Compromising photos of boss emailed
>>    1..3
>>ok 1 - B
>>    1..2
>>    ok 1 - We have lift off!
>>    not ok 2 - Houston, we have a problem
>>not ok 2 - A
>>1..2
>Or (but not recommended):
>not ok 1 - A
>>ok 2 - B
>>1..2
>The latter is not recommended because you're throwing away information, but it should parse the same way as the previous ones with an older TAP parser because ok/not ok binds to the beginning of the line (and thus indented TAP is ignored).
>
>
>In fact, you could put the 1..2 plans before the tests if you wanted to (also called a "leading plan").
>
>
>The reason the nested TAP behavior isn't well-specified is because there were many arguments over it at conferences and hackathons, and then one day (I was in Norway, I think), I got tired of the arguments and just sat down and implemented a mostly backwards-compatible solution. Implementation trumps opinion, so I won.
>
>
>Another thing to note about TAP is that we still don't have structured diagnostics. Again, this was largely because no one could agree on what they should look like. Some wanted producers to be able to output them in whatever format they wanted (JSON, YAML, XML, and so on). I argued that was madness and having a single standard would make parsing much easier. Putting that aside, there were also arguments about how and what information should be captured in said diagnostics. Filename and line number are clear, but what about tests for which that information doesn't make sense? Or what about a timestamp? Local time? GMT (useless)? UTC?
>
>
>So there's lots of work to be done and lots of behavior to be nailed down. Fortunately, TAP's been going strong since the 1980s, but it's only been the past decade or so since it's been named and formally developed.
>
>
>Ask any questions you like.
>
>
>Best,
>Ovid
>--
>IT consulting, training, international recruiting
>       http://www.allaroundtheworld.fr/.
>Buy my book! - http://bit.ly/beginning_perl
>Live and work overseas - http://www.overseas-exile.com/
>
>
>
>
>
>
>On Monday, 11 August 2014, 23:31, Andrew de Andrade <andrew@deandrade.com.br> wrote:
> 
>
>>
>>
>>Yes, this all helps a bunch. I've been using substack's tape (https://github.com/substack/tape) and isaacs' tap (https://github.com/isaacs/node-tap) modules thus far. 
>>
>>
>>tape currently supports v13, which as you pointed out Bruno, does not formally specify how nesting should work, and it does support defining tests in a nested way, but does not report them as nested. Instead, it simply updates the plan count and continues with everything flat/unindented and reports the plan count at the end. For example this code...
>>
>>
>>```
>>var test = require('tape');
>>
>>
>>test('foobar', function(t) {
>>  t.plan(3);
>>  //var test = tape.createHarness();
>>
>>
>>  t.ok(true, 'foo');
>>  t.ok(true, 'bar');
>>
>>
>>  t.test('fibble', function(tt) {
>>    tt.plan(2);
>>    tt.ok(true, 'wobble');
>>    tt.ok(true, 'wizzle');
>>  });
>>});
>>```
>>...produces...
>>
>>
>>```
>>TAP version 13
>># foobar
>>ok 1 foo
>>ok 2 bar
>># fibble
>>ok 3 wobble
>>ok 4 wizzle
>>
>>
>>1..4
>># tests 4
>># pass  4
>>
>>
>># ok
>>```
>>
>>
>>What I did just find is an implementation from Ovid from earlier this year in March:
>>
>>
>>http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
>>
>>https://github.com/Ovid/tap-stream
>>
>>
>>
>>This stream approach is really nice, but the only thing that leaves me scratching my head is that is implies that asynchrony is permitted (parallelized tests, aggregation, out-of-order reporting), but I'm not exactly sure what the implications for this are for the aggregate tap producer. 
>>
>>
>>Should the aggregate reporter report subtests as they are received or should it keep the subtest results that finish out of order in a buffer and always report the tests in a deterministic way? I did see this in the readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that it is left up to the TAP aggregator consuming the stream to either report things in order with or without numbers (if desired) or out of order, but un-numbered. 
>>
>>
>>Given my intended use case, buffering results so that results can always be reported numbered in order would be desirable, since I want to be able to diff TAP results across test runs under different conditions (e.g. different dependency versions). Has anyone done any work with diffing TAP results?
>>
>>Bruno, is tap4j purely synchronous or at least in order or can subtests be asynchronous and reported out of order?
>>
>>
>>Seeing that Ovid is still actively working on this stuff (comments on tap on perlqa3 as recent as April this year), is it save to assume that there are ideas here that will make it into version 14? The test blocks you pointed to does reference version 14 in the example -->  https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks
>>
>>
>>
>>_______________________________________________
>>tap mailing list
>>tap@ietf.org
>>https://www.ietf.org/mailman/listinfo/tap
>>
>>
>>
>
>