Re: [sip-clf] A syslog approach to sip logging

"Spencer Dawkins" <> Wed, 03 February 2010 16:49 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 05CF23A6962 for <>; Wed, 3 Feb 2010 08:49:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.176
X-Spam-Status: No, score=-2.176 tagged_above=-999 required=5 tests=[AWL=-0.178, BAYES_00=-2.599, J_CHICKENPOX_31=0.6, STOX_REPLY_TYPE=0.001]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id NrPd2K-8usLz for <>; Wed, 3 Feb 2010 08:49:56 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id 10A493A68E7 for <>; Wed, 3 Feb 2010 08:49:56 -0800 (PST)
Received: from S73602b ( []) by (node=mrus1) with ESMTP (Nemesis) id 0M5MVr-1NxpqE2xvr-00z82B; Wed, 03 Feb 2010 11:50:00 -0500
Message-ID: <>
From: "Spencer Dawkins" <>
To: "Rainer Gerhards" <>, <>
References: <>
Date: Wed, 3 Feb 2010 10:49:44 -0600
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5843
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.5579
X-Provags-ID: V01U2FsdGVkX1+2ZYlJDD1mW7vNGIXbnsCTKBiRMlLF5bkSiWo yUTBt2NLeYgfnav+dSsWZnOcXE0AlyNbi+EogGETjWxhYAEsuD NbSxsx6rGb7vg1cZIwnCMKb0nGKUm32ylYtinUjdsw=
Subject: Re: [sip-clf] A syslog approach to sip logging
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: SIP Common Log File format discussion list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 03 Feb 2010 16:49:57 -0000



Welcome, and thanks for subscribing!



Your note is very interesting to me - I just fired off a question to Dave 
on-list that might also be helpful for you to think about. I'd appreciate 
any thoughts you can provide!




> Hi all,
> I just subscribed to the sip-clf mailing list. I am the author of rsyslog,
> one of the major open source syslogd's as well as the designer for a 
> number
> of Windows tools that are syslog-based. I have also worked on the IETF 
> syslog
> standardization effort.
> David has made me aware of the current discussion and I am currently 
> working
> through the mailing list. Two things that I would like to comment on are
> transmission of Apache logs via syslog and syslog performance.
> As of my experience, it is quite common to transport Apache clf "files" 
> via
> syslog. There are two was to do this: one is to make apache log in 
> real-time
> to the syslogd, usually with the help of logger or a similar system tool.
> This requires proper engineering and can potentially cause notable
> performance degradation. As I know from the rsyslog user base, these 
> problems
> can be solved and this mode is used in practice, even for high-performance
> sites.
> The other approach is to let apache write to text files and then transfer
> these text files in near-realtime to a syslogd. That is, a process grabs 
> data
> as it is appended to the text log. In rsyslog, the omfile module has
> specifically been written for that use case and, if I remember correctly, 
> the
> root cause for its implementation was Apache clf transfer.
> It may also be worth noting that in the Apache scenario log4j syslog 
> logging
> seems to come together with clf - but I don't have insight if this is true
> for the majority of cases.
> On syslog performance: I have read that expected message volume was
> considered problematic for the syslog use case. It may be worth noting 
> that
> high-volume sites log data via syslog. This may be clf, but the larger ISP 
> or
> financial institutions (or other service providers) already have lots of 
> log
> data that is to be processed. For rsyslog, I know of deployments that 
> average
> 50,000+ messages per second on a single receiving machine. In lab setup, a
> single instance of rsyslog can currently process up to 250,000 msgs per
> second, with this rates going up. The Windows products I am responsible 
> for
> reach similar or higher message rates. Of course, these number depend much 
> on
> the length of the message, parsing overhead and what the final destination
> does with the messages (it is a big difference writing them to a flat 
> ascii
> file or a database and complex filtering also reduces the throughput). 
> Note
> that there is large demand for even faster syslog implementations, which
> leads me to believe that transmission of mass data via syslog is often
> desired.
> I am not sure what message rates are expected for sip and where the actual
> problem for syslog was envisioned. If you have some more information on 
> that,
> it would definitely help me understand the situation at large.
> Rainer Gerhards
> _______________________________________________
> sip-clf mailing list