Re: [tap] RFC Status?

Andrew de Andrade <andrew@deandrade.com.br> Tue, 12 August 2014 00:00 UTC

Return-Path: <andrew@deandrade.com.br>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1BA1B1A0728 for <tap@ietfa.amsl.com>; Mon, 11 Aug 2014 17:00:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 2.102
X-Spam-Level: **
X-Spam-Status: No, score=2.102 tagged_above=-999 required=5 tests=[BAYES_50=0.8, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, J_CHICKENPOX_31=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_NEUTRAL=0.779] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ktDyjo-fWBFG for <tap@ietfa.amsl.com>; Mon, 11 Aug 2014 16:59:57 -0700 (PDT)
Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B9A3E1A06D9 for <tap@ietf.org>; Mon, 11 Aug 2014 16:59:56 -0700 (PDT)
Received: by mail-qc0-f182.google.com with SMTP id i8so2265888qcq.13 for <tap@ietf.org>; Mon, 11 Aug 2014 16:59:55 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=Wo+xWgTgAfgzNdbbXok+S3pbx1RVghuEdT9G2Nzgq9Y=; b=dsDPM+Sz8cs4nIL5mcF6EX06wWVPIfcgPjyiyOPs+mlAITxPx5W0nLsbbZdahAwEL7 MMlrM4ApqQdH9L6kXgkLGAnSVarrxbr0QPYVMraEljIoEveTIGYo76HK11R9kPC4b/IW wKOCHsqu0UxKXyebnBrV3ADIal+pOflIkefkRT2xDINmZNwIhoJUW1XXHg7WLEurwdUi 6AMlNyJa6wskKuApFMO66eQGMyWZHdLKtRpkrP9jZRs2QAOHzzCfnZzDJFeABkXU70RM rfcPCynYfSc9g7o3FBFJE9/ixB+HpJT8OD1Mh243hdpmUxpIQqUkoiBig+aNJJ70LRuQ egaQ==
X-Gm-Message-State: ALoCoQkOYFTnl2CQ1l/rMtAnupjaXnCeVLZ/Wpb1ePC6ZsqhN2P6wVvVU6JYYid/oWFw35LlIt52
X-Received: by 10.224.15.72 with SMTP id j8mr1527423qaa.8.1407801595631; Mon, 11 Aug 2014 16:59:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.51.243 with HTTP; Mon, 11 Aug 2014 16:59:35 -0700 (PDT)
X-Originating-IP: [208.66.27.18]
In-Reply-To: <1407796876.49402.YahooMailNeo@web163504.mail.gq1.yahoo.com>
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <7250AD76-F5DF-494C-8A00-B4320053EBE7@ggvaidya.com> <1407788544.42539.YahooMailNeo@web163506.mail.gq1.yahoo.com> <CAP4gcswppT=HEy4ew-TejcvJjWT8AwXXRjnG2ECZa7tWnbT7ZQ@mail.gmail.com> <1407796876.49402.YahooMailNeo@web163504.mail.gq1.yahoo.com>
From: Andrew de Andrade <andrew@deandrade.com.br>
Date: Mon, 11 Aug 2014 16:59:35 -0700
Message-ID: <CAP4gcsznSZh0gNCUBv5FnhWV4nmC2Z6QUeWN48_cSwUpJy7pYg@mail.gmail.com>
To: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
Content-Type: multipart/alternative; boundary=047d7bea3184060f220500635a6b
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/Yj-tm3O9THNYwfgOjO46NHvXRTw
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Aug 2014 00:00:00 -0000

On Mon, Aug 11, 2014 at 3:41 PM, Bruno P. Kinoshita <
brunodepaulak@yahoo.com.br> wrote:

> > Yes, this all helps a bunch. I've been using substack's tape (
> https://github.com/substack/tape) and isaacs' tap (
> https://github.com/isaacs/node-tap) modules thus far.
>
> I've seen some users reporting issues in the tap-plugin for Jenkins where
> they were using node-tap. Hadn't heard about tape before.
>

If you have links to those issues, that would be most appreciated, since
I'll report them to isaacs. These days I'm using only substack/tape since
I'm doing mostly browser testing. Right now, I'm focused on aggregating the
test results. Once I try out a CI solution like Strider, Jenkins or Travis,
I planned on using tap. If you send me those issues, I might be able to
patch them once I get around to using tap with a CI server.


>
> > What I did just find is an implementation from Ovid from earlier this
> year in March:
> >
> >
> http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
> https://github.com/Ovid/tap-stream
>
> I liked Ovid's implementation. One thing that i have added to my TODO list
> is to look at his examples and use some of them as regression tests in
> tap4j. I think it goes along with your previous suggestion of a set of
> tests for TAP (as in JSON?).
>

Yeah, standardized test suites are immensely useful. I'm not entirely
certain how it would work in this case of testing TAP compliancy though. I
guess JSON is one option (but wouldn't we then be defining yet another test
protocol?). JSON is the logical approach since it would mean that the TAP
examples server as tests for the JSON and vice versa, and all languages
have mature libraries for consuming JSON and working with JSON.

Even if there isn't a clear way to automate checking if an implementation
is correct (since the tests for each would have to be written in the
language of each implementation), just having a corpus of known valid
examples, would be immensely useful.


> > Seeing that Ovid is still actively working on this stuff (comments on
> tap on perlqa3 as recent as April this year), is it save to assume that
> there are ideas here that will make it into version 14? The test blocks you
> pointed to does reference version 14 in the example -->
> https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks
>
> I'm not sure. Do you know if Ovid is in this list? Maybe we should check
> with him if he has interest in joining the thread and help crafting TAP 14
> too?
>
>

I emailed him at the email listed on his github profile. I reckon if he is
interested (or still subscribed via a different email address he's used in
the past), he'll participate directly.