Re: [tap] Thinking big

Michael Peters <mpeters@plusthree.com> Wed, 10 March 2010 20:18 UTC

Return-Path: <mpeters@plusthree.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B6C263A69CE for <tap@core3.amsl.com>; Wed, 10 Mar 2010 12:18:27 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.07
X-Spam-Level:
X-Spam-Status: No, score=-0.07 tagged_above=-999 required=5 tests=[AWL=-0.071, BAYES_50=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yK9VQCU-JGzH for <tap@core3.amsl.com>; Wed, 10 Mar 2010 12:18:26 -0800 (PST)
Received: from mail.plusthree.com (mail.plusthree.com [70.42.42.197]) by core3.amsl.com (Postfix) with ESMTP id 0946C3A6915 for <tap@ietf.org>; Wed, 10 Mar 2010 12:18:25 -0800 (PST)
Received: from [10.0.0.2] (dsl093-241-113.ral1.dsl.speakeasy.net [66.93.241.113]) (authenticated bits=0) by mail.plusthree.com (8.14.2/8.14.2) with ESMTP id o2AKDZH5005649 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 10 Mar 2010 15:13:36 -0500
Message-ID: <4B97FE54.5040307@plusthree.com>
Date: Wed, 10 Mar 2010 15:17:24 -0500
From: Michael Peters <mpeters@plusthree.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100301 Fedora/3.0.3-1.fc11 Thunderbird/3.0.3
MIME-Version: 1.0
To: Salve J Nilsen <sjn@pvv.org>
References: <411886.95806.qm@web65707.mail.ac4.yahoo.com> <D9F21163-EC5A-4023-BD5D-9FCF60A86FBE@hexten.net> <91241.45851.qm@web65716.mail.ac4.yahoo.com> <alpine.DEB.1.00.1003102033060.2167@decibel.pvv.ntnu.no> <4B97FAA8.8090300@plusthree.com> <alpine.DEB.1.00.1003102103550.2167@decibel.pvv.ntnu.no>
In-Reply-To: <alpine.DEB.1.00.1003102103550.2167@decibel.pvv.ntnu.no>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Cc: tap@ietf.org
Subject: Re: [tap] Thinking big
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Mar 2010 20:18:27 -0000

On 03/10/2010 03:08 PM, Salve J Nilsen wrote:

> Right. Are there any specific features in the TAP stream you're missing?
> (E.g. system information, uname(1) output, timestamps, perl version,
> module versions, ..)

There are some things that could be done to remove some of what a TAP 
archive does, but not all of it, or at least without turning TAP into 
some kind of monster.

+ Nested TAP could let us aggregate multiple TAP streams into a single 
TAP stream to represent a single test suite run. But then you aren't 
dealing with the TAP as the test spit it out. Instead as some 
preprocessor assembled it.

+ TAP meta data could be added to a tap stream to record things like 
uname, timing, etc. But unless this is done on a nested TAP stream that 
represents a full test run, then this information only captures an 
individual stream's experience.

+ Extra files (eg, Apache log files): Where would you put them? Trying 
to somehow include them in the TAP stream seems like a really bad idea. 
You'd need some sort of encode/decode so they don't clash with other 
stuff and then they become completely unreadable.

> Or approaching from another directions: Do you know of any very useful
> features available some continuous integration frameworks (or similar)
> that could me made better or possible with the right kind of
> environment/meta information from each single test run?

If you create a TAP archive with extra files then you can already use 
this in Smolder. Upload it and then you can browse those extra files 
when looking at failed tests. We use it at $work a lot.

-- 
Michael Peters
Plus Three, LP