[Sipping] Alternate CLF syntax proposal

[Adam Roach <[e-mail]>]

In the spirit of "send text," I've put together a straw-man proposal for 
an easy-to-generate and fast-to-process extensible format for saving SIP 
log messages:

http://www.ietf.org/internet-drafts/draft-roach-sipping-clf-syntax-00.txt

As an example of the processing that can be performed on this format: 
consider that I have a large file (on the order of 1 GB of data), with 
1,232,896 records in it (to choose a nice, round number). I'd like to 
extract all the information about messages with a particular "From" value.

With a text-based format, I'll be reading and parsing 1,262,485,504 
bytes (every byte in the file) in order to find delimiters.

With the format proposed in this document, I can open the file and then 
do the following about 1,232,896 times:

  - Read 4 bytes (total record length)
  - Fseek 32 bytes to reach the "To Value" pointer and length
  - Read 4 bytes
  - Fseek according to those 4 bytes to the literal value of the to 
header field
  - Read the to header field (let's imagine it's 20 bytes)
  - Fseek to the next record (according to the total record length)

In total, I'm reading 28 bytes per record 1,232,896 times, for a grand 
total of 34,521,088 bytes -- or about 2.7% as much data as I do with a 
text file.

When you're dealing with terabytes of log data, this can make the 
difference between taking one minute to sift data and taking 37 minutes 
to do the same operation. And, of course, it has the advantage that you 
can add more (tagged) data to each record without causing any additional 
processing load.

/a