Re: [sip-ops] [dispatch] SIP-CLF: Results on ASCII vs. binary representation

[Dean Willis <[e-mail]>]

On May 4, 2009, at 2:58 PM, Adam Roach wrote:

> Dean Willis wrote:
>> Right. In my experience, binary offers only marginal performance  
>> over structured ASCII.
>
> I'll correct this slightly: binary offers only marginal performance  
> over *properly* structured ASCII. I argue that the Apache log format  
> doesn't qualify for this description. (I say this after having to  
> deal with processing relatively large Apache logs, and suffering  
> from the vast quantities of time such processing takes).
>
> I'm okay with Tom's suggestion to add ASCII length (and tag?) fields  
> to the text format. If people want to take it that direction, I  
> could re-cast my proposed format so it contains fixed-length ASCII  
> number representations for pointers, lengths, and tags; and so that  
> it terminates records with something like an ASCII 0x0D.


One could design things such that each record contains an initial  
"structure" filed with a fixed-format  series of indices (total  
length, number of fields, lengths for each field, followed by  
variable--length data fields, tab (or other reserved char) delimited  
followed by a CRLF terminator (included in total-length index). This  
gives us records that are fast-parseable, but still human-readable.

--
Dean