Re: [tap] RFC Status?

Andrew de Andrade <andrew@deandrade.com.br> Mon, 11 August 2014 21:31 UTC

Return-Path: <andrew@deandrade.com.br>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 651E11A00BE for <tap@ietfa.amsl.com>; Mon, 11 Aug 2014 14:31:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.301
X-Spam-Level: *
X-Spam-Status: No, score=1.301 tagged_above=-999 required=5 tests=[BAYES_20=-0.001, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, J_CHICKENPOX_31=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_NEUTRAL=0.779] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2a2U5ioJlMQp for <tap@ietfa.amsl.com>; Mon, 11 Aug 2014 14:31:18 -0700 (PDT)
Received: from mail-qa0-f41.google.com (mail-qa0-f41.google.com [209.85.216.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D8F341A006C for <tap@ietf.org>; Mon, 11 Aug 2014 14:31:17 -0700 (PDT)
Received: by mail-qa0-f41.google.com with SMTP id j7so8438228qaq.28 for <tap@ietf.org>; Mon, 11 Aug 2014 14:31:16 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=1+5YzKLBUZe9YU+PmvKWizZlh03RHd0lAMZDeeZf8v4=; b=jrZtzkXGGXaWk0UTuzZr3Quh+q88g3Hj0L+YnByDJ++azuX9QIA33tKn0nAY3oyFx/ 9tKVkOrXB3C2l5XHkQhCe318bl4yjtCoj/bPM2AeGV7xA+IIN+qGf0YBP5URGeESuqd+ iJ20SQGXLvWF5P55Yv7vq+FlbGSfJ4cFLxQSBFqGKh3Al1+T+8z0O63X4YPmKux57rpw PzzxJwn1pi5Xz/TEdZOnqBR+UvkVWn9hhuVUY0Vky5qu5jT7EQlAMmdCqvIKsA2MxgTO UmZFxi0J2vqyzGBOw1ntrXLSG1WcsPj/MusJLWhtskj01kZwZQu2cocr2OTshmo8dFy6 A2OQ==
X-Gm-Message-State: ALoCoQnjSiKyrh0UaT7syqNGDM4PiR9InfAuq8avoz8l8Mfp80okom+RA9YXBdlGAbfsVG/6Tt4p
X-Received: by 10.224.15.72 with SMTP id j8mr662829qaa.8.1407792676806; Mon, 11 Aug 2014 14:31:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.51.243 with HTTP; Mon, 11 Aug 2014 14:30:56 -0700 (PDT)
X-Originating-IP: [208.66.27.18]
In-Reply-To: <1407788544.42539.YahooMailNeo@web163506.mail.gq1.yahoo.com>
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <7250AD76-F5DF-494C-8A00-B4320053EBE7@ggvaidya.com> <1407788544.42539.YahooMailNeo@web163506.mail.gq1.yahoo.com>
From: Andrew de Andrade <andrew@deandrade.com.br>
Date: Mon, 11 Aug 2014 14:30:56 -0700
Message-ID: <CAP4gcswppT=HEy4ew-TejcvJjWT8AwXXRjnG2ECZa7tWnbT7ZQ@mail.gmail.com>
To: "Bruno P. Kinoshita" <brunodepaulak@yahoo.com.br>
Content-Type: multipart/alternative; boundary=047d7bea31846ba1200500614661
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/UtWHHa6bmLzXhnM_OHPnl6Q2MTs
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 11 Aug 2014 21:31:20 -0000

Yes, this all helps a bunch. I've been using substack's tape (
https://github.com/substack/tape) and isaacs' tap (
https://github.com/isaacs/node-tap) modules thus far.

tape currently supports v13, which as you pointed out Bruno, does not
formally specify how nesting should work, and it does support defining
tests in a nested way, but does not report them as nested. Instead, it
simply updates the plan count and continues with everything flat/unindented
and reports the plan count at the end. For example this code...

```
var test = require('tape');

test('foobar', function(t) {
  t.plan(3);
  //var test = tape.createHarness();

  t.ok(true, 'foo');
  t.ok(true, 'bar');

  t.test('fibble', function(tt) {
    tt.plan(2);
    tt.ok(true, 'wobble');
    tt.ok(true, 'wizzle');
  });
});
```
...produces...

```
TAP version 13
# foobar
ok 1 foo
ok 2 bar
# fibble
ok 3 wobble
ok 4 wizzle

1..4
# tests 4
# pass  4

# ok
```

What I did just find is an implementation from Ovid from earlier this year
in March:

http://blogs.perl.org/users/ovid/2014/03/perlqa-hackathon-in-lyon---day-1.html
https://github.com/Ovid/tap-stream

This stream approach is really nice, but the only thing that leaves me
scratching my head is that is implies that asynchrony is permitted
(parallelized tests, aggregation, out-of-order reporting), but I'm not
exactly sure what the implications for this are for the aggregate tap
producer.

Should the aggregate reporter report subtests as they are received or
should it keep the subtest results that finish out of order in a buffer and
always report the tests in a deterministic way? I did see this in the
readme, https://github.com/Ovid/tap-stream#caveats , implying AFAICT that
it is left up to the TAP aggregator consuming the stream to either report
things in order with or without numbers (if desired) or out of order, but
un-numbered.

Given my intended use case, buffering results so that results can always be
reported numbered in order would be desirable, since I want to be able to
diff TAP results across test runs under different conditions (e.g.
different dependency versions). Has anyone done any work with diffing TAP
results?

Bruno, is tap4j purely synchronous or at least in order or can subtests be
asynchronous and reported out of order?

Seeing that Ovid is still actively working on this stuff (comments on tap
on perlqa3 as recent as April this year), is it save to assume that there
are ideas here that will make it into version 14? The test blocks you
pointed to does reference version 14 in the example -->
https://web.archive.org/web/20110611083916/http://testanything.org/wiki/index.php/Test_Blocks