Re: [tap] Valids subtests (was: RFC Status?)

Andrew Rodland <andrew@cleverdomain.org> Sat, 20 September 2014 00:01 UTC

Return-Path: <andrew@cleverdomain.org>
X-Original-To: tap@ietfa.amsl.com
Delivered-To: tap@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9849B1A88FA for <tap@ietfa.amsl.com>; Fri, 19 Sep 2014 17:01:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.278
X-Spam-Level:
X-Spam-Status: No, score=-1.278 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5HxMEXMLSHQs for <tap@ietfa.amsl.com>; Fri, 19 Sep 2014 17:01:17 -0700 (PDT)
Received: from smtp.pobox.com (smtp.pobox.com [208.72.237.35]) by ietfa.amsl.com (Postfix) with ESMTP id 260311A02EB for <tap@ietf.org>; Fri, 19 Sep 2014 17:01:16 -0700 (PDT)
Received: from smtp.pobox.com (unknown [127.0.0.1]) by pb-smtp0.pobox.com (Postfix) with ESMTP id E47903CC39 for <tap@ietf.org>; Fri, 19 Sep 2014 20:01:15 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc :content-type; s=sasl; bh=FzFaC96ts3OxdPdSerammSlt8lc=; b=jNB3+b EUdXoMMIGv6eTJu9EyRaulIJO0d8UqLDG/16mJ0mLyeCLHciq7j7RJVBray9XnW9 F/6rf3Vle5WDcNRHNxM0+YHYmpYGoBLS2hRSNMG3jEpv9uFZa8oOI7zangreehgW H5rd6yWKns6EzLIgMV7uZSwp8ar2365dw9aBA=
Received: from pb-smtp0. (unknown [127.0.0.1]) by pb-smtp0.pobox.com (Postfix) with ESMTP id DB7813CC38 for <tap@ietf.org>; Fri, 19 Sep 2014 20:01:15 -0400 (EDT)
Received: from mail-ig0-f170.google.com (unknown [209.85.213.170]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by pb-smtp0.pobox.com (Postfix) with ESMTPSA id 7B1343CC36 for <tap@ietf.org>; Fri, 19 Sep 2014 20:01:15 -0400 (EDT)
Received: by mail-ig0-f170.google.com with SMTP id a13so1682529igq.3 for <tap@ietf.org>; Fri, 19 Sep 2014 17:01:14 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.42.40.13 with SMTP id j13mr4182979ice.50.1411171274950; Fri, 19 Sep 2014 17:01:14 -0700 (PDT)
Received: by 10.107.36.16 with HTTP; Fri, 19 Sep 2014 17:01:14 -0700 (PDT)
In-Reply-To: <1411114619.77920.YahooMailNeo@web126104.mail.ne1.yahoo.com>
References: <CAP4gcszybVr5Hw3mg=uTi8tqpA3wEVwo=zf2876RWhy_CmozZw@mail.gmail.com> <CAHhgV8jr6ZnsfUkpFC4OL0AwRX-aen7v-7KjcN3e0_19s7steg@mail.gmail.com> <1410980929.82809.YahooMailNeo@web126106.mail.ne1.yahoo.com> <1410981416.81953.YahooMailNeo@web163504.mail.gq1.yahoo.com> <CABFQKmObv0EG=iVhzPNYGKwK44k7QtEV+PfrvBnDjn35MjbzBg@mail.gmail.com> <1411114619.77920.YahooMailNeo@web126104.mail.ne1.yahoo.com>
Date: Fri, 19 Sep 2014 20:01:14 -0400
Message-ID: <CABFQKmOe4hY5wrSnt6ArEz6bjPpHAaf9MSO_mtHn8Yt92e-T4w@mail.gmail.com>
From: Andrew Rodland <andrew@cleverdomain.org>
To: Ovid <curtis_ovid_poe@yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1
X-Pobox-Relay-ID: 3D777C22-4059-11E4-A403-BD2DC4D60FE0-16769786!pb-smtp0.pobox.com
Archived-At: http://mailarchive.ietf.org/arch/msg/tap/kYIJ1FMla9nWDTnWQ35ri0X4cOY
Cc: "tap@ietf.org" <tap@ietf.org>
Subject: Re: [tap] Valids subtests (was: RFC Status?)
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap/>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 20 Sep 2014 00:01:21 -0000

On Fri, Sep 19, 2014 at 4:16 AM, Ovid <curtis_ovid_poe@yahoo.com> wrote:
> Hi Andrew,
>
> In that, we agreed that it was desirable that the determination of whether a
> line was junk/unknown wasn't context sensitive.
>
> I've tried searching for where I said that, but I can't find it. Can you
> provide me with the quote so I can understand where the confusion lies? I
> may very well be mistaken here.
>

=== begin quote (http://www.ietf.org/mail-archive/web/tap/current/msg00480.html)

That's a very annoying edge case you have there, but I would argue
that anything which *could* be parsed as a TAP line MUST be parsed as
such, thus rendering the above invalid. Otherwise, we'd have no idea
if you were spitting out bad TAP or something else was spitting out
junk which happened to look like TAP.

Thus:

* If something *can* be parsed as a TAP line, do so, even if it
renders the TAP document invalid.
* If something *can't* be parsed as a TAP line, ignore it.

=== end quote

> I fully admit that subtests were never considered in the original
> formulation of junk lines, but I think my description of this as being valid
> and the eight space indented line being an unknown (and therefore discarded)
> line is correct:
>
>> ok 1 - foo
>>        ok 1 - bar indented too far
>> ok 2 - bar passed
>
> The reasoning works like this: historically it has always been the case that
> ok/not ok status is bound to the beginning of the string and even a single
> space in front of ok/not ok invalid. With a subtest, we make an exception if
> an only if it is indented exactly four spaces (in the context of the
> containing scope) and is parseable as TAP. Otherwise, writing a parser could
> be a nightmare. Consider:
>
>> ok
>> not ok
>> ok
>> 1..3
>
> Most people aren't aware of this, but that's a perfectly valid (but failing)
> TAP stream. While it's not recommended, the test numbers have always been
> optional. Here's the relevant line from the TAP grammar:
>
>  test ::= status positiveInteger? description? directive?
>
> Because TAP is designed to be both human-readable and computer-parseable,
> there exists the unfortunate problem that sometimes it's just a touch harder
> to parse than normal and some of the plans for it might make life more
> difficult. For example, if we followed your suggestion, we would then have
> this scenario:
>
>> ok
>>        ok this is working now
>> ok bar passed
>> 1..2
>
> As currently described (by my understanding), the above is valid because the
> "ok this is working now" line is indented too far and thus discarded. This
> makes parsing much simpler. However, you argue that the above is invalid
> because the "ok" line is valid TAP in an invalid context. That makes sense,
> but consider the plans for structured diagnostics:
>
>> not ok # TODO darn it!
>>     ---
>>     file     : t/foo.t
>>     datetime : 2015-01-23 01:23:45
>>     have     : 7
>>     want     : 8
>>     comment  : |
>>                 We still have an off by one error and it's not
>>                 ok that we still haven't tracked it down
>>     ...
>> ok bar passed
>> 1..2
>
> Parsers are not required to parse structured diagnostics (this allows older
> parsers to ignore them), but that last line of the comment above could be
> considered valid TAP in an invalid context and result in a parse error, even
> though the above is perfectly fine.
>

Yes, I see the conflict here. I could suggest that structured
diagnostics be indented *two* spaces relative to the test that they
annotate, which would eliminate any potential for confusion since a
valid test is going to be indented 4n spaces. I'm not sure if that's
too esoteric.