Re: [tap] Valids subtests (was: RFC Status?)

Ovid <curtis_ovid_poe@yahoo.com> Wed, 17 September 2014 20:18 UTC

Return-Path: <curtis_ovid_poe@yahoo.com>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 10AA41A6ED9 for <tap@ietfa.amsl.com>; Wed, 17 Sep 2014 13:18:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.049
X-Spam-Level:
X-Spam-Status: No, score=-3.049 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_31=0.6, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-1.652, SPF_PASS=-0.001, URIBL_DBL_ABUSE_REDIR=0.001, URIBL_DBL_REDIR=0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mpz_FABmbwJq for <tap@ietfa.amsl.com>; Wed, 17 Sep 2014 13:18:04 -0700 (PDT)
Received: from nm29-vm1.bullet.mail.ne1.yahoo.com (nm29-vm1.bullet.mail.ne1.yahoo.com [98.138.90.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 640811A212A for <tap@ietf.org>; Wed, 17 Sep 2014 13:18:04 -0700 (PDT)
Received: from [98.138.226.176] by nm29.bullet.mail.ne1.yahoo.com with NNFMP; 17 Sep 2014 20:18:03 -0000
Received: from [98.138.89.249] by tm11.bullet.mail.ne1.yahoo.com with NNFMP; 17 Sep 2014 20:18:03 -0000
Received: from [127.0.0.1] by omp1041.mail.ne1.yahoo.com with NNFMP; 17 Sep 2014 20:18:03 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 565767.43204.bm@omp1041.mail.ne1.yahoo.com
Received: (qmail 57328 invoked by uid 60001); 17 Sep 2014 20:18:03 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1410985083; bh=taLAt5ABfLCswmdH8ynGPqXyau33o2foz/QFm6SKXew=; h=References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ORjieaz/POjlnn/TFLf5wxNzIUk6aXL5ty8GK8PjBOMq9GiWkF1ars4NLPWUWVjOiSoNZaYBfAf2vCQzeLYYUhqYNK7wcxUdRsEm0AONgBtoiO5lbi7m5gy8IupQm8Qe54OVPo5UwVDz1hnVBno0MN+ha8Y5NvyW5zH+F7oRPjs=
X-YMail-OSG: jPRtlQgVM1nrmykp80l4UxfY8FKaVw_6sX7pRbRwz0_NDTD RVCHzeCxvl8T8JsM5ltlWBGfm5XuCmb8_l.qAbv6OTbGuTMmrKORu53RBQra 8NGEDXtojjVYa7.TD9bdZxMc_cUmCLPseHHNBxmCILurwAW8NNFkDsNnHoXM Ib5iJV_kPk93cv4Oab66eno77CvrAy18_UESGOC8AUu9lO4RQ_VgBfgMt6Wm M5qB7PcluSK2zCkjAVEtboYN8Tj3rvXUH2Vb8FzpW8GSegdRs.s5_oFv8_eP 0bEWMf2n1EHFgqr9R4PjRP8FMOLvs_8HaHT6sz8Q_yauTwkOdz1YP9LQWJHv M0ewrygQqOYBOHhR3bZUx1VAKvHnSVmJWJnt0xXWXTorCdf2tAbuM55QvLOD fHrezkWGND2B_FbSX_.tZJJw01ESHIxR0MYgnDQ55_afVORrz2pkwnzB.MgI 0rcf13tzZ5Z3f8OP4.mMy_u6lShf2VE8fdi62EUx7EOLjploTr1cr9gGo3zn yVoYf7cvZ5K1uprRS7haIW9Wts8lrhy7MHnIjjIf9EQCMtRvJaihFM0ctnm9 2rNgLjyHtkYMoMWPh2GG6mR07unIbEdqdAoiPOlu.CtR01fwwdD2qZ4unJrO AdxLGa07AKrtoHaNYkWOHtmEiXb6igbdBRor0paMbR7NMfkY8Yjo6CiwJJCn JIGypPL25lCW5bVFDI26K2aQps_lgBZz50CILji1Y8NdYbOu1zT5g_apSdr3 N42TDgPoFrQHAp2H8hpj_fu0iwDBUfcW4pS8_yfGisLpnr3V2lyVx8c6QDsE J1bJi9mVDfq7_1an1XzW9p62LpI_FMW7HnTmk2nx1JlMcrCjXO2Y16NCIoV5 4lzQvdMyC8.vdnXvQu_CdB70_LhvQ.QVx0twffeQpMYm6BFygdqkc452gci. WbEOsHQ0gy2JQlCIwZNAVyjlPkD8I05cehw--
Received: from [2.6.227.184] by web126106.mail.ne1.yahoo.com via HTTP; Wed, 17 Sep 2014 13:18:03 PDT
X-Rocket-MIMEInfo: 002.001, U2VlIG15IGdyYW1tYXIgaW4gYW5vdGhlciB0aHJlYWQuIEEgc3VidGVzdCBpcyBpbmRlbnRlZCBmb3Igc3BhY2VzLCBlaXRoZXIgYmVnaW5zIG9yIGVuZHMgd2l0aCBhIHBsYW4gKGFuZCBjb3VsZCBzdGFydCB3aXRoIGFuIG9wdGlvbmFsIHZlcnNpb24gbnVtYmVyKSBhbmQgdGhlIHRlc3QgbGluZSBpbW1lZGlhdGVseSBmb2xsb3dpbmcgdGhlIHN1YnRlc3QgbXVzdCBoYXZlIGFuIG9rL25vdCBvayBzdGF0dXMgdG8gc3VtbWFyaXplIHRoZSBwYXNzL2ZhaWwgc3RhdHVzIG9mIHRoZSBzdWJ0ZXN0LiBUaGF0IGUBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.203.696
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <CAHhgV8jr6ZnsfUkpFC4OL0AwRX-aen7v-7KjcN3e0_19s7steg@mail.gmail.com> <1410980929.82809.YahooMailNeo@web126106.mail.ne1.yahoo.com> <1410981416.81953.YahooMailNeo@web163504.mail.gq1.yahoo.com> <1410982831.46047.YahooMailNeo@web126105.mail.ne1.yahoo.com> <1410984733.67023.YahooMailNeo@web163504.mail.gq1.yahoo.com>
Message-ID: <1410985083.80803.YahooMailNeo@web126106.mail.ne1.yahoo.com>
Date: Wed, 17 Sep 2014 13:18:03 -0700
From: Ovid <curtis_ovid_poe@yahoo.com>
To: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
In-Reply-To: <1410984733.67023.YahooMailNeo@web163504.mail.gq1.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="1181703601-1529448496-1410985083=:80803"
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/w1J2_UncPAEEHIvURsQTGxpU49Q
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] Valids subtests (was: RFC Status?)
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: Ovid <curtis_ovid_poe@yahoo.com>
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Sep 2014 20:18:12 -0000

See my grammar in another thread. A subtest is indented for spaces, either begins or ends with a plan (and could start with an optional version number) and the test line immediately following the subtest must have an ok/not ok status to summarize the pass/fail status of the subtest. That ensures that even if older parsers can't recognize the subtest, they'll correctly pick up the summary line and still produce valid results. 
 
Best,
Ovid
--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas - http://www.overseas-exile.com/


On Wednesday, 17 September 2014, 22:12, Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> wrote:
 

>
>
>Thanks Ovid!
>
>
>Probably when we start discussing subtests I'll chip in with many questions. As, am I allowed to have subtests before and after test results?
>
>
>1..2
>ok 1 - foo
>    not ok 1 - some text
>
>ok 2 - bar passed
>    not ok 1 - another text
>
>
>
>1..2
>    not ok 1 - some text
>
>ok 1 - foo
>    not ok 1 - another text
>
>ok 2 - bar passed
>    not ok 1 - another text
>
>
>Because my first guess was that subtests would always come after test elements, and be linked to the previous element. 
>
>
>Thanks again
>Bruno
>
>
>
>>________________________________
>> From: Ovid <curtis_ovid_poe@yahoo.com>
>>To: Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> 
>>Cc: "tap@ietf.org" <tap@ietf.org> 
>>Sent: Wednesday, September 17, 2014 4:40 PM
>>Subject: Re: Valids subtests (was: [tap] RFC Status?)
>> 
>>
>>
>>Bruno,
>>
>>
>>Yes, those are perfectly valid subtests. I, like an idiot, forgot to show the subtest plans in my examples. 
>> 
>>Best,
>>Ovid
>>--
>>IT consulting, training, international recruiting
>>       http://www.allaroundtheworld.fr/.
>>Buy my book! - http://bit.ly/beginning_perl
>>Live and work overseas -
 http://www.overseas-exile.com/
>>
>>
>>
>>On Wednesday, 17 September 2014, 21:16, Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> wrote:
>> 
>>
>>>
>>>
>>>Ovid, 
>>>
>>>
>>>Would a subtest as follows be valid?
>>>
>>>
>>>    ok 1 - subtest 1a
>>>    ok 2 - subtest 1b
>>>    1..2
>>>ok 1 - Subtest 1
>>>    ok 1 - subtest 2a
>>>    ok 2 - subtest 2b
>>>    1..2
>>>ok 2 - Subtest 2
>>>1..2
>>>ok
>>>
>>>
>>>A user posted this TAP stream generated with Test::More to tap4j [1]. I'll probably end up writing a custom parser for it to support streams generated by Perl tools, but I'd like to know whether it would be considered valid or not.
>>>
>>>
>>>TIA
>>>Bruno
>>>
>>>
>>>
>>>
>>>[1] https://github.com/tupilabs/tap4j/issues/15
>>>
>>>
>>>
>>>
>>>
>>>>________________________________
>>>> From: Ovid <publiustemp-tapx@yahoo.com>
>>>>To: Leon Timmermans <fawaka@gmail.com>om>; Andrew de Andrade <andrew@deandrade.com.br> 
>>>>Cc: "tap@ietf.org" <tap@ietf.org> 
>>>>Sent: Wednesday, September 17, 2014 4:08 PM
>>>>Subject: Re: [tap] RFC Status?
>>>> 
>>>>
>>>>
>>>>A few notes on this.
>>>>
>>>>
>>>>For subtests, they are indented four spaces. Always four spaces. If they are indented eight spaces, you know they're a subtest of a subtest (assuming there's an intervening subtest. That means this is valid TAP, but there is no subtest because it's indented too far:
>>>>ok 1 - foo
>>>>>        ok 1 - bar indented too far
>>>>>ok 2 - bar passed
>>>>The "ok 1 - bar" line would be discarded as "unknown". 
>>>>
>>>>
>>>>So if, in a stream of TAP, you find something which would be valid tap if it were unindented by four spaces, it's a subtest. The trailing, unindented summary line must mirror the status of the subtest. Thus, the following is invalid:
>>>>
>>>>
>>>>ok 1 - foo
>>>>>    not ok 1 - bar indented too far
>>>>>ok 2 - bar passed
>>>>>
>>>>>Currently, Test::Harness treats the above as passing when it should not. I believe Schwern filed a bug for that. I know Andy Armstrong and I both took a quick swing at extending Test::Harness but discovered that it was more difficult than it appeared.
>>>>
>>>>
>>>>The grammar would be modified to look sort of like this (modifying http://search.cpan.org/dist/Test-Harness/lib/TAP/Parser/Grammar.pm):
>>>>
>>>>line           ::= (comment | test | subtest | unknown | bailout ) "\n"
>>>>>subtest        ::= subtest_line { subtest_line } test
>>>>>subtest_line   ::= '    ' line
>>>>That assumes we're OK with a recursive grammar. Also, the grammar doesn't really have a way of identifying the semantics of the trailing line on the subtest grammar line.
>>>>
>>>>
>>>>If you have other questions about subtests, I think I can give definitive answers.
>>>>
>>>>
>>>>As for the YAMLish structured diagnostics, those were never official, as far as I recall. There were several disputes, one of which was whether or not we should use YAML or JSON (I recall even XML being mentioned). Some argued that any structured diagnostic should be allowed, but I reject that firmly because that makes life hell for the parser. If you asked me what my stance is today, I would think that structured diagnostics should valid JSON, but that raises questions about whitespace issues, UTF-8, and so on. If we could agree on the format, then we'd need to agree on the keys in the structured diagnostic dictionary (and how to extend them).
>>>>
>>>>
>>>>Best,
>>>>Ovid
>>>>--
>>>>IT consulting, training, international recruiting
>>>>       http://www.allaroundtheworld.fr/.
>>>>Buy
 my book! - http://bit.ly/beginning_perl
>>>>Live and work overseas -
 http://www.overseas-exile.com/
>>>>
>>>>
>>>>
>>>>On Wednesday, 17 September 2014, 20:07, Leon Timmermans <fawaka@gmail.com> wrote:
>>>> 
>>>>
>>>>>
>>>>>
>>>>>On Mon, Aug 11, 2014 at 8:42 PM, Andrew de Andrade <andrew@deandrade.com.br> wrote:
>>>>>
>>>>>I'm working on some testing libraries for JavaScript based on TAP. Specifically, I'm trying to explore making it possible for library authors to not only check the results of their TAP tests, but also map-reduce the TAP results of the dependents of their library to see how changes to their library impact the correctness of their dependents. 
>>>>>>
>>>>>>
>>>>>>e.g. I am the author of library A. I publish version 0.1.0. Another author includes my library as a dependency in library B and in library C and uses TAP to test those libraries. Given this scenario, it would be nice if I could, when modifying my library, run not only my tests, but also run the tests of all my dependents built with my previous version and my current version and compare how my changes impacted the "correctness" of their programs. 
>>>>>>
>>>>>>
>>>>>>Beyond the benefits to a community, helping authors know when they can likely safely upgrade their deps, this type of runner would also help in an corporate environment with lots of code re-use between teams. 
>>>>>>
>>>>>>
>>>>>>While thinking about this, I decided to go out and figure out if there is such a thing as nested TAP in the protocol specification. While searching for this, I came across this group, which is a great, but it appears there has been no activity here in a while and it looks like all the wiki pages with prior information about the state of the TAP protocol becoming an IETF RFC have disappeared. 
>>>>>>
>>>>>>
>>>>>>What's the current state of TAP? Is there still interest in taking this to RFC status? I found some previous discussions on the list about nested TAP. Did those make it into the specification in any way? If so, where can I find examples of correct nested tap results?
>>>>>>
>>>>>>
>>>>>>Furthermore, is there a standardized set of test fixtures for the current version that any implementation can be tested against for correctness and performance? Having worked a bit with JSON schema, I found that one of the most useful tools was a standardized set of tests to check if a particular implementation conforms to the protocol standard. (see: https://github.com/json-schema/JSON-Schema-Test-Suite )
>>>>>>
>>>>>>
>>>>>I've been recently writing a TAP Harness, and for subtests I followed what Perl has been doing for years (just like TAP::Stream): have indented subtests follow by a non-indented summary line. None of this is described in detail anywhere though :-/, nor is the interaction with the barely defined YAMLish well-explored AFAIK.
>>>>>
>>>>>Leon
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>tap mailing list
>>>>>tap@ietf.org
>>>>>https://www.ietf.org/mailman/listinfo/tap
>>>>>
>>>>>
>>>>>
>>>>
>>>>_______________________________________________
>>>>tap mailing list
>>>>tap@ietf.org
>>>>https://www.ietf.org/mailman/listinfo/tap
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>