Re: [tap] Thinking big

Salve J Nilsen <sjn@pvv.org> Wed, 10 March 2010 21:58 UTC

Return-Path: <sjn@pvv.ntnu.no>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id ACD0F3A69D1 for <tap@core3.amsl.com>; Wed, 10 Mar 2010 13:58:34 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.58
X-Spam-Level:
X-Spam-Status: No, score=-2.58 tagged_above=-999 required=5 tests=[AWL=0.020, BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ys8ga4QS95JZ for <tap@core3.amsl.com>; Wed, 10 Mar 2010 13:58:33 -0800 (PST)
Received: from decibel.pvv.ntnu.no (decibel.pvv.ntnu.no [IPv6:2001:700:300:1900::1:2]) by core3.amsl.com (Postfix) with ESMTP id 7A8D53A68EE for <tap@ietf.org>; Wed, 10 Mar 2010 13:58:33 -0800 (PST)
Received: from sjn by decibel.pvv.ntnu.no with local (Exim 4.69) (envelope-from <sjn@pvv.ntnu.no>) id 1NpTvE-0002Eo-T9; Wed, 10 Mar 2010 22:58:32 +0100
Date: Wed, 10 Mar 2010 22:58:32 +0100 (CET)
From: Salve J Nilsen <sjn@pvv.org>
X-X-Sender: sjn@decibel.pvv.ntnu.no
To: Michael Peters <mpeters@plusthree.com>
In-Reply-To: <4B97FE54.5040307@plusthree.com>
Message-ID: <alpine.DEB.1.00.1003102157180.2167@decibel.pvv.ntnu.no>
References: <411886.95806.qm@web65707.mail.ac4.yahoo.com> <D9F21163-EC5A-4023-BD5D-9FCF60A86FBE@hexten.net> <91241.45851.qm@web65716.mail.ac4.yahoo.com> <alpine.DEB.1.00.1003102033060.2167@decibel.pvv.ntnu.no> <4B97FAA8.8090300@plusthree.com> <alpine.DEB.1.00.1003102103550.2167@decibel.pvv.ntnu.no> <4B97FE54.5040307@plusthree.com>
User-Agent: Alpine 1.00 (DEB 882 2007-12-20)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: tap@ietf.org
Subject: Re: [tap] Thinking big
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Mar 2010 21:58:34 -0000

Michael Peters said:
> On 03/10/2010 03:08 PM, Salve J Nilsen wrote:
>
>> Right. Are there any specific features in the TAP stream you're 
>> missing? (E.g. system information, uname(1) output, timestamps, 
>> perl version, module versions, ..)
>
> There are some things that could be done to remove some of what a 
> TAP archive does, but not all of it, or at least without turning TAP 
> into some kind of monster.
>
> + Nested TAP could let us aggregate multiple TAP streams into a 
> single TAP stream to represent a single test suite run. But then you 
> aren't dealing with the TAP as the test spit it out. Instead as some 
> preprocessor assembled it.
>
> + TAP meta data could be added to a tap stream to record things like 
> uname, timing, etc. But unless this is done on a nested TAP stream 
> that represents a full test run, then this information only captures 
> an individual stream's experience.

Couldn't the same tap stream "linking" be achieved just by assigning a 
"test run id" that is consistent across streams in the same test run?


>> Or approaching from another directions: Do you know of any very 
>> useful features available some continuous integration frameworks 
>> (or similar) that could me made better or possible with the right 
>> kind of environment/meta information from each single test run?
>
> If you create a TAP archive with extra files then you can already 
> use this in Smolder. Upload it and then you can browse those extra 
> files when looking at failed tests. We use it at $work a lot.

I'm asking more about what _can't_ be done with Smolder, and if some 
new aggregatable information could make it easier to implement these 
features at some point. E.g. test run timing, system load at end of 
run, I/O usage, etc. Once you have a few months of collected data you 
can figure out new and non-obvious things.


- Salve

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <sjn@foo.no>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)