Re: [tap] Thinking big

Salve J Nilsen <> Wed, 10 March 2010 21:58 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id ACD0F3A69D1 for <>; Wed, 10 Mar 2010 13:58:34 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.58
X-Spam-Status: No, score=-2.58 tagged_above=-999 required=5 tests=[AWL=0.020, BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id Ys8ga4QS95JZ for <>; Wed, 10 Mar 2010 13:58:33 -0800 (PST)
Received: from ( [IPv6:2001:700:300:1900::1:2]) by (Postfix) with ESMTP id 7A8D53A68EE for <>; Wed, 10 Mar 2010 13:58:33 -0800 (PST)
Received: from sjn by with local (Exim 4.69) (envelope-from <>) id 1NpTvE-0002Eo-T9; Wed, 10 Mar 2010 22:58:32 +0100
Date: Wed, 10 Mar 2010 22:58:32 +0100 (CET)
From: Salve J Nilsen <>
To: Michael Peters <>
In-Reply-To: <>
Message-ID: <>
References: <> <> <> <> <> <> <>
User-Agent: Alpine 1.00 (DEB 882 2007-12-20)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Subject: Re: [tap] Thinking big
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 10 Mar 2010 21:58:34 -0000

Michael Peters said:
> On 03/10/2010 03:08 PM, Salve J Nilsen wrote:
>> Right. Are there any specific features in the TAP stream you're 
>> missing? (E.g. system information, uname(1) output, timestamps, 
>> perl version, module versions, ..)
> There are some things that could be done to remove some of what a 
> TAP archive does, but not all of it, or at least without turning TAP 
> into some kind of monster.
> + Nested TAP could let us aggregate multiple TAP streams into a 
> single TAP stream to represent a single test suite run. But then you 
> aren't dealing with the TAP as the test spit it out. Instead as some 
> preprocessor assembled it.
> + TAP meta data could be added to a tap stream to record things like 
> uname, timing, etc. But unless this is done on a nested TAP stream 
> that represents a full test run, then this information only captures 
> an individual stream's experience.

Couldn't the same tap stream "linking" be achieved just by assigning a 
"test run id" that is consistent across streams in the same test run?

>> Or approaching from another directions: Do you know of any very 
>> useful features available some continuous integration frameworks 
>> (or similar) that could me made better or possible with the right 
>> kind of environment/meta information from each single test run?
> If you create a TAP archive with extra files then you can already 
> use this in Smolder. Upload it and then you can browse those extra 
> files when looking at failed tests. We use it at $work a lot.

I'm asking more about what _can't_ be done with Smolder, and if some 
new aggregatable information could make it easier to implement these 
features at some point. E.g. test run timing, system load at end of 
run, I/O usage, etc. Once you have a few months of collected data you 
can figure out new and non-obvious things.

- Salve

sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)