Re: [tap] [tap-l] questions regarding draft standard compliance

EBo <ebo@users.sourceforge.net> Fri, 27 August 2010 14:44 UTC

Return-Path: <ebo@users.sourceforge.net>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 188183A6954 for <tap@core3.amsl.com>; Fri, 27 Aug 2010 07:44:09 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -99.999
X-Spam-Level:
X-Spam-Status: No, score=-99.999 tagged_above=-999 required=5 tests=[BAYES_50=0.001, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3LlJ7TCaiMC3 for <tap@core3.amsl.com>; Fri, 27 Aug 2010 07:44:07 -0700 (PDT)
Received: from raba-115.swcp.com (raba-115.swcp.com [198.59.115.131]) by core3.amsl.com (Postfix) with ESMTP id 9A9873A6980 for <tap@ietf.org>; Fri, 27 Aug 2010 07:44:04 -0700 (PDT)
Received: from panda.swcp.com (panda.swcp.com [216.184.2.211]) by raba-115.swcp.com (8.13.6/8.13.6) with ESMTP id o7REiZ00063016; Fri, 27 Aug 2010 08:44:35 -0600 (MDT) (envelope-from ebo@users.sourceforge.net)
Received: from panda.swcp.com (localhost [127.0.0.1]) by panda.swcp.com (8.14.0/8.14.0) with ESMTP id o7REiUUh011750; Fri, 27 Aug 2010 08:44:30 -0600
Received: (from apache@localhost) by panda.swcp.com (8.14.0/8.14.0/Submit) id o7REiU75011748; Fri, 27 Aug 2010 08:44:30 -0600
X-Authentication-Warning: panda.swcp.com: apache set sender to ebo@users.sourceforge.net using -f
To: Ovid <publiustemp-tapx@yahoo.com>
MIME-Version: 1.0
Received: from ip98-183-93-214.lf.br.cox.net (ip98-183-93-214.lf.br.cox.net [98.183.93.214]) by roundtest.swcp.com with HTTP/1.1 (POST); Fri, 27 Aug 2010 08:44:30 -0600
Date: Fri, 27 Aug 2010 08:44:30 -0600
From: EBo <ebo@users.sourceforge.net>
Organization: SourceForge
In-Reply-To: <95075.76292.qm@web65706.mail.ac4.yahoo.com>
References: <6a5c279351b0c4804bdc73d1a1346f87@swcp.com> <95075.76292.qm@web65706.mail.ac4.yahoo.com>
Message-ID: <711145d2ff54ab6508950ae5278763de@swcp.com>
X-Sender: ebo@users.sourceforge.net
User-Agent: RoundCube Webmail/0.3.1
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
X-Mailman-Approved-At: Mon, 30 Aug 2010 16:25:27 -0700
Cc: Test Anything Protocol <tap-l@testanything.org>, tap@ietf.org
Subject: Re: [tap] [tap-l] questions regarding draft standard compliance
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: ebo@users.sourceforge.net
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 27 Aug 2010 15:24:40 -0000

>> 2) Should I report here, to tap-l, discrepancies from  Test-Harness
>> divinations from the draft standard?  Or should I take that  off-list
to
>> the
>> maintainers?  The case in point is a descriptive  string in
Test-Harness
>> which is not Safe-String compliant.  Normally, I  would take this
>> directly
>> to the maintainers, but it is my understanding that  Test-Harness is
the
>> exemplar for the IETF draft standard, and thought these  issues fodder
>> for
>> general discussion.
> 
> I don't mind seeing the discussion here, though there's also an IETF 
> list: tap@ietf.org

Thanks Ovid, I have CC'ed tap@ietf.org to solicit additional comments on
their end.

>> 1) is  the draft standard for the SKIP and TODO directives to be  case
>> sensitive?
> 
> They should not be case-sensitive.  As a general rule of thumb (there
are 
> exceptions), if the standard deviates from the behaviour of
Test::Harness,
> it 
> means the standard needs to be corrected.  This is because we have, on
> several 
> occasions, discovered that the official Test::Harness for Perl has, for
> years, 
> permitted unexpected behaviour.  Thus, we tend to say that the code is
the 
> actual standard in order to avoid breaking backwards-compatibility.

OK.  I would have thought that the IETF standard would be an attempt to
repair oddities in future versions of Test::Harness than to document it as
canonical source.  Otherwise why would they explicitly set up a Safe-String
in their grammar if you can include escape \# (see test 36 from
t/source_handler.t).  Fortunately in 11,000+ test results, only one appears
to violate this IETF grammar rule.

On the IETF side, what are your thoughts on this matter?

> In this case, you can look at the code for the grammar:
> 
>  
http://cpansearch.perl.org/src/ANDYA/Test-Harness-3.22/lib/TAP/Parser/Grammar.pm
> 
> 
> And you should note the following:
> 
>                 if ($desc =~ m/^ ( [^\\\#]* (?: \\. [^\\\#]* )* )
>                        \# \s* (SKIP|TODO) \b \s* (.*) $/ix
>                   )
>                 {
>                     ( $desc, $dir, $explanation ) = ( $1, $2, $3 );
>                 }
> 
> That clearly shows that the directives are case-insensitive ("TodO" is
> allowed, 
> for example) and even the space between the hash (#) and the directive
is 
> optional (though it's interesting that no space is required after.  "not
> ok 1 
> #TodO-whee!" is a valid TODO test. I think I wrote that regex, so this
> might be 
> my fault).

The optional space is a minor issue and can be easily accounted for by
correcting the IETF standard, but I was more concerned by the Safe-String
rule violation.

On the IETF side, how do you want to deal with SP in the grammar?

> Still, it's better to ask if you're in doubt.

Thanks and best regards,

  EBo --