[sip-clf] New CLF Syntax draft (text with index)

Adam Roach <adam@nostrum.com> Thu, 07 May 2009 18:55 UTC

Return-Path: <adam@nostrum.com>
X-Original-To: sip-clf@core3.amsl.com
Delivered-To: sip-clf@core3.amsl.com
Received: from localhost (localhost []) by core3.amsl.com (Postfix) with ESMTP id 2560B3A6B6C for <sip-clf@core3.amsl.com>; Thu, 7 May 2009 11:55:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.564
X-Spam-Status: No, score=-2.564 tagged_above=-999 required=5 tests=[AWL=0.036, BAYES_00=-2.599, SPF_PASS=-0.001]
Received: from mail.ietf.org ([]) by localhost (core3.amsl.com []) (amavisd-new, port 10024) with ESMTP id qYelQIKSTO-Y for <sip-clf@core3.amsl.com>; Thu, 7 May 2009 11:54:59 -0700 (PDT)
Received: from nostrum.com (nostrum-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:267::2]) by core3.amsl.com (Postfix) with ESMTP id 409FF3A68EA for <sip-clf@ietf.org>; Thu, 7 May 2009 11:54:58 -0700 (PDT)
Received: from [] (vicuna-alt.estacado.net []) (authenticated bits=0) by nostrum.com (8.14.3/8.14.3) with ESMTP id n47IuNIT072511 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 7 May 2009 13:56:23 -0500 (CDT) (envelope-from adam@nostrum.com)
Message-ID: <4A032ED7.7030504@nostrum.com>
Date: Thu, 07 May 2009 13:56:23 -0500
From: Adam Roach <adam@nostrum.com>
User-Agent: Postbox 1.0b11 (Macintosh/2009041623)
MIME-Version: 1.0
To: sip-clf@ietf.org
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Received-SPF: pass (nostrum.com: is authenticated by a trusted mechanism)
Subject: [sip-clf] New CLF Syntax draft (text with index)
X-BeenThere: sip-clf@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: SIP Common Log File format discussion list <sip-clf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/sip-clf>, <mailto:sip-clf-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/sip-clf>
List-Post: <mailto:sip-clf@ietf.org>
List-Help: <mailto:sip-clf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sip-clf>, <mailto:sip-clf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 07 May 2009 18:55:00 -0000

This version defines a text format in which each record is composed of 
two lines in a log file. The first line is primarily pointers into the 
second line. The second line contains the actual logged fields, 
separated by tab characters.

This approach retains the fast-search capabilities that I've been 
advocating, while allowing the use of simple, text-based tools, as Vijay 
has been promoting. This hybrid approach does come with a slight 
increase in log file size; for example, writing out the same 100,000 log 
entries in each of the three formats proposed so far:

   - Text:   25 Mb
   - Binary: 29 Mb
   - Hybrid: 37 Mb

This is about a 20% premium over the binary format, and a 48% premium 
over the text format. Speed of generation should be on the same order of 
speed as the other two versions, and speed of processing should be 
approximately the same as processing binary.

The new version of the document can be found here: