Re: [tap] RFC Status?

Leon Timmermans <fawaka@gmail.com> Sat, 20 September 2014 22:10 UTC

Return-Path: <fawaka@gmail.com>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 763DC1A02BA for <tap@ietfa.amsl.com>; Sat, 20 Sep 2014 15:10:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.701
X-Spam-Level:
X-Spam-Status: No, score=0.701 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xRjHzuMhNmJG for <tap@ietfa.amsl.com>; Sat, 20 Sep 2014 15:10:11 -0700 (PDT)
Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B56E31A02D0 for <tap@ietf.org>; Sat, 20 Sep 2014 15:10:09 -0700 (PDT)
Received: by mail-ie0-f179.google.com with SMTP id tp5so652329ieb.24 for <tap@ietf.org>; Sat, 20 Sep 2014 15:10:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=nUsCAzfWdDCevaXlrkL0dqh2QDWZEBDuxSjMIriypeE=; b=KmTzbRAjcmnV6lRdR+PvFxYoh4egpQLbV3lMFiC3o8Vd37wn2iHXoC8dwt9T0gRv6s bhuu2LkYYQxKN89v4qaIBrLTRMoMV3GW8DptUwCXFXqb+7Yfe8yn9KZqoVw6pl7etxD1 BAx/a0Dj9kcTIwCHhchRPlpdvRP3Yv3jQ31lhS2iRrGPvxo65pVU9IiJfsQ2o43t21Fe WLrJUyIR3bmcgxyofeuKqha99KwRhFmhDgTB38gN4zd9FLVpd5NSMRioggzyRzGMUqHj 5jhTYCzp9tbEN1CDgI/EoG3d7/Lmc361VuQVPy30YSI9RFaolM2YyiCy1zucw5dE4OvF xosg==
X-Received: by 10.50.62.50 with SMTP id v18mr5079877igr.21.1411251009048; Sat, 20 Sep 2014 15:10:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.50.109.198 with HTTP; Sat, 20 Sep 2014 15:09:48 -0700 (PDT)
In-Reply-To: <1410980929.82809.YahooMailNeo@web126106.mail.ne1.yahoo.com>
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <CAHhgV8jr6ZnsfUkpFC4OL0AwRX-aen7v-7KjcN3e0_19s7steg@mail.gmail.com> <1410980929.82809.YahooMailNeo@web126106.mail.ne1.yahoo.com>
From: Leon Timmermans <fawaka@gmail.com>
Date: Sun, 21 Sep 2014 00:09:48 +0200
Message-ID: <CAHhgV8hUG3xPvY_=zRJ2wKfCc5iPA4OSVw=yxTnof6_v6HHTmQ@mail.gmail.com>
To: Ovid <publiustemp-tapx@yahoo.com>
Content-Type: multipart/alternative; boundary=047d7bdc085415619b0503867b4e
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/S4K2ikQDgtn4IoO5ILABerzE-pE
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] RFC Status?
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 20 Sep 2014 22:10:13 -0000

On Wed, Sep 17, 2014 at 9:08 PM, Ovid <publiustemp-tapx@yahoo.com> wrote:

> A few notes on this.
>
> For subtests, they are indented four spaces. Always four spaces. If they
> are indented eight spaces, you know they're a subtest of a subtest
> (assuming there's an intervening subtest. That means this is valid TAP, but
> there is no subtest because it's indented too far:
>
> That is very useful to know, makes life for the consumer much easier.

> ok 1 - foo
>         ok 1 - bar indented too far
> ok 2 - bar passed
>
> The "ok 1 - bar" line would be discarded as "unknown".
>

I think I agree, though one could argue that's an unknown subline of a
subtest.


> So if, in a stream of TAP, you find something which would be valid tap if
> it were unindented by four spaces, it's a subtest. The trailing,
> unindented summary line must mirror the status of the subtest. Thus, the
> following is invalid:
>
> ok 1 - foo
>     not ok 1 - bar indented too far
> ok 2 - bar passed
>
> Currently, Test::Harness <http://search.cpan.org/dist/Test-Harness/>
> treats the above as passing when it should not. I believe Schwern filed a
> bug for that. I know Andy Armstrong and I both took a quick swing at
> extending Test::Harness but discovered that it was more difficult than it
> appeared.
>

I'm currently giving a parse error in that condition :-)

As for the YAMLish structured diagnostics, those were never official, as
> far as I recall. There were several disputes, one of which was whether or
> not we should use YAML or JSON (I recall even XML being mentioned). Some
> argued that any structured diagnostic should be allowed, but I reject that
> firmly because that makes life hell for the parser. If you asked me what my
> stance is today, I would think that structured diagnostics should valid
> JSON, but that raises questions about whitespace issues, UTF-8, and so on.
> If we could agree on the format, then we'd need to agree on the keys in the
> structured diagnostic dictionary (and how to extend them).
>

Deciding on the keys is bikeshedding, deciding on the format decidedly
isn't. To be honest I'm not even sure what would be good or bad at this
stage, though I can agree with not accepting every format and their cousin.

Leon