Re: [tap] RFC Status?

Ovid <curtis_ovid_poe@yahoo.com> Tue, 12 August 2014 07:26 UTC

Return-Path: <curtis_ovid_poe@yahoo.com>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EDEB51A076F for <tap@ietfa.amsl.com>; Tue, 12 Aug 2014 00:26:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.635
X-Spam-Level:
X-Spam-Status: No, score=0.635 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_31=0.6, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-0.668, SPF_PASS=-0.001, URIBL_DBL_ABUSE_REDIR=0.001, URIBL_DBL_REDIR=0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IsuCxfBS4hIG for <tap@ietfa.amsl.com>; Tue, 12 Aug 2014 00:26:19 -0700 (PDT)
Received: from nm3-vm5.bullet.mail.ne1.yahoo.com (nm3-vm5.bullet.mail.ne1.yahoo.com [98.138.91.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 829AE1A075F for <tap@ietf.org>; Tue, 12 Aug 2014 00:26:19 -0700 (PDT)
Received: from [98.138.100.118] by nm3.bullet.mail.ne1.yahoo.com with NNFMP; 12 Aug 2014 07:26:18 -0000
Received: from [98.138.88.237] by tm109.bullet.mail.ne1.yahoo.com with NNFMP; 12 Aug 2014 07:26:18 -0000
Received: from [127.0.0.1] by omp1037.mail.ne1.yahoo.com with NNFMP; 12 Aug 2014 07:26:18 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 820128.87005.bm@omp1037.mail.ne1.yahoo.com
Received: (qmail 92581 invoked by uid 60001); 12 Aug 2014 07:26:18 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1407828378; bh=9suL571bHwi+gWcWIVfhq3ftgNuYmL1UrmX9BGeWe6A=; h=References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=KpnJh0yFEs8EyZTzUnRiGfKkM+grY1NyWjQdG10oLfpqgIywNAQfd73xSSIadFP1XZ7DAlhEr6KEUQpREAvwYXBuq/iQ6J+SvWv3dkw7AAaetSl3qtqm64VzIasCgqb9/mBlpnlUVSyADvGjyB6bEit4vePJpttQseiCGXetZPo=
X-YMail-OSG: lzmCxqQVM1l1JD8EZOJ5tgcwce5pnwBn2XjqgTfvYlckDTf 0LdaYyMhCYn4hrKiVMCT5WANofHNFekG9bJCu9dfWWNkr_PBH2Ap6qMZ5NKE 1701z6KZkz7CDI5xKFGilMFvABJvbpVtfAPSgfaLO3TH0FDRATH539zr4q2. u.g9Zik.xQ6pYw1YDLxGoNXQMi5bJSrJd4qXm_.SR2OlTNz3WThjgFVUsGh7 F1wio_M_q3f54AsqnXVU124OoiiqdEd213yrCbs3BOvrhMzBimvaqc7iXgt_ 2Eez0l5Eprs.PsX26ZvvLksNvkgZnEzghvxPllcNkRaE4z5UZMfarVjvCrjm S7_e7uqhl3rZXj8c49DgRO4AVJ.JdfKifs87GYXC3DLopX_qjG7g1NUsWlvt uSJ7pz017Mg8cAf_gX2swWImN9UDL8q07Yr8GKNnbNlDIEwf5mj8xL9ifBlU 94KujbJNChdWmosdQlb4HBgRPeXLepc13vrD6d8d5AiBU_ntXB5GcPeWlcVY VNh0cUrN.qq9teLpNOs.DWLj5CG_rUGLMVo1gAjvxWtPh6UhOyh4zmleDOaX 7FMQlI3dLTROyE4aKtJPAG.3kVrJJKu5Zj_KNEZvDCmurcW8d3BgVJKpbzv3 Dhj5W5vUZbKuq3Ja2Aaye0.w3GR9a47Axoz57ZhYWotbTlysf3WXPRHqJ67e kQ9bR43f6FpjtvcOqpMrnTW3W7KQ8aLGx1hJedlh1k0NX5dCF.eQtJOv_hp. 5v._LjPqWrZ.ChrsDzv3hzW8Ce7VdVH6_TiLVW7SmW3FdjaQijpz8Tx5Oc8h QqwX_icKzFTqlD3I9UzIJkzSC.KxyJa0Bf2a6uWt6EO59sZ6v_q6WYfKEWwd TC9.cCamrcWriQkEXzbicrbCMB9kjw5wAINXaeRYY1cqhQkWz9yi9kvPGUOO zEF0dU1EirND_VVPSV5LuGefXvuadI7WMGwfUiorVmD1hV0U6paUGiD5N6Vf FdnOzto9V3nSrzMxIQDJhKQe7Z1PtWLzHZIK5wGeRPBj99ymMRUpYDS2buzm BldCPn1zdm7PFDqWlEkkdzRJQrqFvUiyT93oYxsvHLWWAWWB.V9jXX.CvZT2 ..OiXHeumqo0RPdo5.Sk7sImlLEOMve7wXc6MyrDpI83kk9rWrRibAz1zJc0 NKYWB3zp_AueguJwXbetsxadI8Zj4
Received: from [86.206.169.249] by web126106.mail.ne1.yahoo.com via HTTP; Tue, 12 Aug 2014 00:26:18 PDT
X-Rocket-MIMEInfo: 002.001, SGkgYWxsLAoKWWVzLCBJJ20gc3RpbGwgb24gdGhpcyBsaXN0LiBJIHdhcyBwbGVhc2FudGx5IHN1cnByaXNlZCBhdCB0aGUgYnVyc3Qgb2YgYWN0aXZpdHkgSSBzYXcgdG9kYXkuCgpJJ2QgbG92ZSB0byBzZWUgYSBUQVAgdjE0LiBNb3JlIGltcG9ydGFudGx5LCBJJ2QgbG92ZSB0byBzZWUgYSBzdGFuZGFyZCBjcmVhdGVkIG91dCBvZiB0aGlzIHNvIHRoYXQgd2UgY291bGQgYWN0dWFsbHkgc2F5ICJsb29rIGF0IHRoZSBSRkMiLgrCoApCZXN0LApPdmlkCi0tCklUIGNvbnN1bHRpbmcsIHRyYWluaW5nLCBpbnQBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.201.700
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <7250AD76-F5DF-494C-8A00-B4320053EBE7@ggvaidya.com> <1407788544.42539.YahooMailNeo@web163506.mail.gq1.yahoo.com> <CAP4gcswppT=HEy4ew-TejcvJjWT8AwXXRjnG2ECZa7tWnbT7ZQ@mail.gmail.com> <1407796876.49402.YahooMailNeo@web163504.mail.gq1.yahoo.com>
Message-ID: <1407828378.66372.YahooMailNeo@web126106.mail.ne1.yahoo.com>
Date: Tue, 12 Aug 2014 00:26:18 -0700
From: Ovid <curtis_ovid_poe@yahoo.com>
To: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>, Andrew de Andrade <andrew@deandrade.com.br>
In-Reply-To: <1407796876.49402.YahooMailNeo@web163504.mail.gq1.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="1181703601-1907520072-1407828378=:66372"
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/qDS05qNXj1L3h6mayr-L41cQwOM
X-Mailman-Approved-At: Tue, 12 Aug 2014 00:57:54 -0700
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: Ovid <curtis_ovid_poe@yahoo.com>
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Aug 2014 07:26:22 -0000

Hi all,

Yes, I'm still on this list. I was pleasantly surprised at the burst of activity I saw today.

I'd love to see a TAP v14. More importantly, I'd love to see a standard created out of this so that we could actually say "look at the RFC".
 
Best,
Ovid
--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas - http://www.overseas-exile.com/


On Tuesday, 12 August 2014, 0:41, Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> wrote:
 

>
>
>> Yes, this all helps a bunch. I've been using substack's tape (https://github.com/substack/tape) and isaacs' tap (https://github.com/isaacs/node-tap) modules thus far. 
>
>
>I've seen some users reporting issues in the tap-plugin for Jenkins where they were using node-tap. Hadn't heard about tape before. 
>
>
>> What I did just find is an implementation from Ovid from earlier this year in March:
>>
>> http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
>https://github.com/Ovid/tap-stream
>
>
>I liked Ovid's implementation. One thing that i have added to my TODO list is to look at his examples and use some of them as regression tests in tap4j. I think it goes along with your previous suggestion of a set of tests for TAP (as in JSON?).
>
>
>> Bruno, is tap4j purely synchronous or at least in order or can subtests be asynchronous and reported out of order?
>
>
>Purely synchronous at the moment. There is a factory, that creates a TapConsumer, which receives a TAP stream or java.util.File, and then it calls the parser.
>
>
>The parser itself uses a state machine (based on SnakeYAML parser). So my guess is that it is not too hard to add a new asynchronous consumer, that passes each line received to the parser, in order. This assync consumer would deal with java threading, and raise any exception received from the parser (such as receiving tokens in the order Test-Result -> Tap-Plan -> Test-Result).
>
>
>> Seeing that Ovid is still actively working on this stuff (comments on tap on perlqa3 as recent as April this year), is it save to assume that there are ideas here that will make it into version 14? The test blocks you pointed to does reference version 14 in the example -->  https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks
>
>
>I'm not sure. Do you know if Ovid is in this list? Maybe we should check with him if he has interest in joining the thread and help crafting TAP 14 too? 
>
>
>Bruno
>
>
>
>>________________________________
>> From: Andrew de Andrade <andrew@deandrade.com.br>
>>To: Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> 
>>Cc: Gaurav Vaidya <gaurav@ggvaidya.com>om>; "tap@ietf.org" <tap@ietf.org> 
>>Sent: Monday, August 11, 2014 6:30 PM
>>Subject: Re: [tap] RFC Status?
>> 
>>
>>
>>Yes, this all helps a bunch. I've been using substack's tape (https://github.com/substack/tape) and isaacs' tap (https://github.com/isaacs/node-tap) modules thus far. 
>>
>>
>>tape currently supports v13, which as you pointed out Bruno, does not formally specify how nesting should work, and it does support defining tests in a nested way, but does not report them as nested. Instead, it simply updates the plan count and continues with everything flat/unindented and reports the plan count at the end. For example this code...
>>
>>
>>```
>>var test = require('tape');
>>
>>
>>test('foobar', function(t) {
>>  t.plan(3);
>>  //var test = tape.createHarness();
>>
>>
>>  t.ok(true, 'foo');
>>  t.ok(true, 'bar');
>>
>>
>>  t.test('fibble', function(tt) {
>>    tt.plan(2);
>>    tt.ok(true, 'wobble');
>>    tt.ok(true, 'wizzle');
>>  });
>>});
>>```
>>...produces...
>>
>>
>>```
>>TAP version 13
>># foobar
>>ok 1 foo
>>ok 2 bar
>># fibble
>>ok 3 wobble
>>ok 4 wizzle
>>
>>
>>1..4
>># tests 4
>># pass  4
>>
>>
>># ok
>>```
>>
>>
>>What I did just find is an implementation from Ovid from earlier this year in March:
>>
>>
>>http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
>>
>>https://github.com/Ovid/tap-stream
>>
>>
>>
>>This stream approach is really nice, but the only thing that leaves me scratching my head is that is implies that asynchrony is permitted (parallelized tests, aggregation, out-of-order reporting), but I'm not exactly sure what the implications for this are for the aggregate tap producer. 
>>
>>
>>Should the aggregate reporter report subtests as they are received or should it keep the subtest results that finish out of order in a buffer and always report the tests in a deterministic way? I did see this in the readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that it is left up to the TAP aggregator consuming the stream to either report things in order with or without numbers (if desired) or out of order, but un-numbered. 
>>
>>
>>Given my intended use case, buffering results so that results can always be reported numbered in order would be desirable, since I want to be able to diff TAP results across test runs under different conditions (e.g. different dependency versions). Has anyone done any work with diffing TAP results?
>>
>>Bruno, is tap4j purely synchronous or at least in order or can subtests be asynchronous and reported out of order?
>>
>>
>>Seeing that Ovid is still actively working on this stuff (comments on tap on perlqa3 as recent as April this year), is it save to assume that there are ideas here that will make it into version 14? The test blocks you pointed to does reference version 14 in the example -->  https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks
>>
>>
>>
>>
>
>_______________________________________________
>tap mailing list
>tap@ietf.org
>https://www.ietf.org/mailman/listinfo/tap
>
>
>