Re: [tap] RFC Status?

"Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br> Mon, 11 August 2014 22:41 UTC

Return-Path: <brunodepaulak@yahoo.com.br>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2CA611A06F2 for <tap@ietfa.amsl.com>; Mon, 11 Aug 2014 15:41:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.948
X-Spam-Level: *
X-Spam-Status: No, score=1.948 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_31=0.6, RCVD_IN_DNSWL_NONE=-0.0001, REPTO_QUOTE_YAHOO=0.646] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AIC9vlA8q6M0 for <tap@ietfa.amsl.com>; Mon, 11 Aug 2014 15:41:19 -0700 (PDT)
Received: from nm24-vm6.bullet.mail.gq1.yahoo.com (nm24-vm6.bullet.mail.gq1.yahoo.com [98.136.217.101]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CF5E41A0658 for <tap@ietf.org>; Mon, 11 Aug 2014 15:41:18 -0700 (PDT)
Received: from [98.137.12.63] by nm24.bullet.mail.gq1.yahoo.com with NNFMP; 11 Aug 2014 22:41:17 -0000
Received: from [98.137.12.207] by tm8.bullet.mail.gq1.yahoo.com with NNFMP; 11 Aug 2014 22:41:16 -0000
Received: from [127.0.0.1] by omp1015.mail.gq1.yahoo.com with NNFMP; 11 Aug 2014 22:41:16 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 947545.24641.bm@omp1015.mail.gq1.yahoo.com
Received: (qmail 32920 invoked by uid 60001); 11 Aug 2014 22:41:16 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.br; s=s1024; t=1407796876; bh=AAkUz3BnRxXi6RM2b3ZbvCTs//vG+qCnMoJkkbLkBMI=; h=References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=HzUhWw/u4rC5BYv3Ti5IdNCdVI9wqv8MGvuvmY8bQMMYiV8UukdRHLWmw9F0ItgRHDLSbHx7QZ7WYih5kwH3f6E9SZi/hMmpuNALjUkxaquFItgfd0+cFsLL7Y2XO14FmBzCfboKEVFBojwM+BUaNKBkIsdtUjfo7SQqr3yCnq0=
X-YMail-OSG: Hn2ulI8VM1m_0Iemqv.R05G1EaCX.xwjJDM1heO130MEhTq mEC0dT6aKJTWw8aguiaEZF.6KVFNYoPz_xctq9WNn7bqdvdu_exZu16qqGQT 1Yane4pk.NsFH3GqYBBL04AY3YWHYM8D_fHDOYvN2k4uyLKufEQ46yicyUz9 O_PkS0__ZChcQfWwmxjLZeNzPKA7sIbdnfoTlcic69KwTLcI7fAQuKqCWaTh 6pjGBfKsqADsK9sM2CviR1EzRtx_Q1LYLyuiIxZYckjcxJjjmJvsxESCDP7j ZpjI6EpNuW7cd1UhKr5qzj_b1J7a0dHvUncurxBU3EgZaaAtuIgE5lTVu6jP wsNBQcsdYYG75hQq8Eua09b9tiWnrCUxI86XojT7YDnMgqRBVVngfbM0wq6Q hSXTlfdYdQYtANJG3lP2SMRlRDctP9pK9zNjTUOqxw5UH.Vt85xp9GzFq.c0 UT4NM7vOYCwqQsoUezDyRE.O9W38S_iIiaoFE6RNfXPL3Toseb6eLqkS0HaU hCvmUlhJnoXmQLftDG9SBLx12OeXmWJV80RMMDSNtxfhEeB2vhIQiQBp31Jp 1ymafEzcw9eoZwhmg92bZdogbRY01oD8DraPn.gKTcR.HQi6vIXjE4HLxrgL tRPByZUZ72IW0DpIZjroo0YWu9ofXRCCwDhmuZ9LnOlvAwA7zrQZoU2j6odu jRGs6qtbpOo1dXvOfE_0.BVygqA5uCylVT5PAwmZ4G5eX782X6z9Lme_wf53 MWq.5gaGC81mB_9f6vFus_Rt_K_di6Cw8PEboSTig5eGCi5F0WyfpXksFvic VAl6ZQCMQqjiewjIFGhlAqCBhKDsfPomZIBjofKruQTpgkX9csDEMQsBe.1B Q7DAITfnRTg5lZsHR7.68ttkARqEmKTP0ZV7CAP88n2YZ1_IT7QjsZvIccBm mKThMiG4U7oHGd_0PdrJrjZ9BvHtWqTIXV19ybQzfPaJvyxDUj7bRSTfY
Received: from [189.120.175.52] by web163504.mail.gq1.yahoo.com via HTTP; Mon, 11 Aug 2014 15:41:16 PDT
X-Rocket-MIMEInfo: 002.001, PsKgWWVzLCB0aGlzIGFsbCBoZWxwcyBhIGJ1bmNoLiBJJ3ZlIGJlZW4gdXNpbmcgc3Vic3RhY2sncyB0YXBlIChodHRwczovL2dpdGh1Yi5jb20vc3Vic3RhY2svdGFwZSkgYW5kIGlzYWFjcycgdGFwIChodHRwczovL2dpdGh1Yi5jb20vaXNhYWNzL25vZGUtdGFwKSBtb2R1bGVzIHRodXMgZmFyLsKgCgpJJ3ZlIHNlZW4gc29tZSB1c2VycyByZXBvcnRpbmcgaXNzdWVzIGluIHRoZSB0YXAtcGx1Z2luIGZvciBKZW5raW5zIHdoZXJlIHRoZXkgd2VyZSB1c2luZyBub2RlLXRhcC4gSGFkbid0IGhlYXJkIGFib3UBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.201.700
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <7250AD76-F5DF-494C-8A00-B4320053EBE7@ggvaidya.com> <1407788544.42539.YahooMailNeo@web163506.mail.gq1.yahoo.com> <CAP4gcswppT=HEy4ew-TejcvJjWT8AwXXRjnG2ECZa7tWnbT7ZQ@mail.gmail.com>
Message-ID: <1407796876.49402.YahooMailNeo@web163504.mail.gq1.yahoo.com>
Date: Mon, 11 Aug 2014 15:41:16 -0700
From: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
To: Andrew de Andrade <andrew@deandrade.com.br>
In-Reply-To: <CAP4gcswppT=HEy4ew-TejcvJjWT8AwXXRjnG2ECZa7tWnbT7ZQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="1159194385-1965410481-1407796876=:49402"
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/JBsUunJ38Bsc3fhwzFd7AIsul5U
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 11 Aug 2014 22:41:21 -0000

> Yes, this all helps a bunch. I've been using substack's tape (https://github.com/substack/tape) and isaacs' tap (https://github.com/isaacs/node-tap) modules thus far. 

I've seen some users reporting issues in the tap-plugin for Jenkins where they were using node-tap. Hadn't heard about tape before. 

> What I did just find is an implementation from Ovid from earlier this year in March:
>
> http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
https://github.com/Ovid/tap-stream

I liked Ovid's implementation. One thing that i have added to my TODO list is to look at his examples and use some of them as regression tests in tap4j. I think it goes along with your previous suggestion of a set of tests for TAP (as in JSON?).

> Bruno, is tap4j purely synchronous or at least in order or can subtests be asynchronous and reported out of order?

Purely synchronous at the moment. There is a factory, that creates a TapConsumer, which receives a TAP stream or java.util.File, and then it calls the parser.

The parser itself uses a state machine (based on SnakeYAML parser). So my guess is that it is not too hard to add a new asynchronous consumer, that passes each line received to the parser, in order. This assync consumer would deal with java threading, and raise any exception received from the parser (such as receiving tokens in the order Test-Result -> Tap-Plan -> Test-Result).

> Seeing that Ovid is still actively working on this stuff (comments on tap on perlqa3 as recent as April this year), is it save to assume that there are ideas here that will make it into version 14? The test blocks you pointed to does reference version 14 in the example -->  https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks

I'm not sure. Do you know if Ovid is in this list? Maybe we should check with him if he has interest in joining the thread and help crafting TAP 14 too? 

Bruno


>________________________________
> From: Andrew de Andrade <andrew@deandrade.com.br>
>To: Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> 
>Cc: Gaurav Vaidya <gaurav@ggvaidya.com>om>; "tap@ietf.org" <tap@ietf.org> 
>Sent: Monday, August 11, 2014 6:30 PM
>Subject: Re: [tap] RFC Status?
> 
>
>
>Yes, this all helps a bunch. I've been using substack's tape (https://github.com/substack/tape) and isaacs' tap (https://github.com/isaacs/node-tap) modules thus far. 
>
>
>tape currently supports v13, which as you pointed out Bruno, does not formally specify how nesting should work, and it does support defining tests in a nested way, but does not report them as nested. Instead, it simply updates the plan count and continues with everything flat/unindented and reports the plan count at the end. For example this code...
>
>
>```
>var test = require('tape');
>
>
>test('foobar', function(t) {
>  t.plan(3);
>  //var test = tape.createHarness();
>
>
>  t.ok(true, 'foo');
>  t.ok(true, 'bar');
>
>
>  t.test('fibble', function(tt) {
>    tt.plan(2);
>    tt.ok(true, 'wobble');
>    tt.ok(true, 'wizzle');
>  });
>});
>```
>...produces...
>
>
>```
>TAP version 13
># foobar
>ok 1 foo
>ok 2 bar
># fibble
>ok 3 wobble
>ok 4 wizzle
>
>
>1..4
># tests 4
># pass  4
>
>
># ok
>```
>
>
>What I did just find is an implementation from Ovid from earlier this year in March:
>
>
>http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
>
>https://github.com/Ovid/tap-stream
>
>
>
>This stream approach is really nice, but the only thing that leaves me scratching my head is that is implies that asynchrony is permitted (parallelized tests, aggregation, out-of-order reporting), but I'm not exactly sure what the implications for this are for the aggregate tap producer. 
>
>
>Should the aggregate reporter report subtests as they are received or should it keep the subtest results that finish out of order in a buffer and always report the tests in a deterministic way? I did see this in the readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that it is left up to the TAP aggregator consuming the stream to either report things in order with or without numbers (if desired) or out of order, but un-numbered. 
>
>
>Given my intended use case, buffering results so that results can always be reported numbered in order would be desirable, since I want to be able to diff TAP results across test runs under different conditions (e.g. different dependency versions). Has anyone done any work with diffing TAP results?
>
>Bruno, is tap4j purely synchronous or at least in order or can subtests be asynchronous and reported out of order?
>
>
>Seeing that Ovid is still actively working on this stuff (comments on tap on perlqa3 as recent as April this year), is it save to assume that there are ideas here that will make it into version 14? The test blocks you pointed to does reference version 14 in the example -->  https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks
>
>
>
>