Re: [sip-clf] A syslog approach to sip logging

["Rainer Gerhards" <[e-mail]>]

Hi all,

I just subscribed to the sip-clf mailing list. I am the author of rsyslog,
one of the major open source syslogd's as well as the designer for a number
of Windows tools that are syslog-based. I have also worked on the IETF syslog
standardization effort.

David has made me aware of the current discussion and I am currently working
through the mailing list. Two things that I would like to comment on are
transmission of Apache logs via syslog and syslog performance.

As of my experience, it is quite common to transport Apache clf "files" via
syslog. There are two was to do this: one is to make apache log in real-time
to the syslogd, usually with the help of logger or a similar system tool.
This requires proper engineering and can potentially cause notable
performance degradation. As I know from the rsyslog user base, these problems
can be solved and this mode is used in practice, even for high-performance
sites.

The other approach is to let apache write to text files and then transfer
these text files in near-realtime to a syslogd. That is, a process grabs data
as it is appended to the text log. In rsyslog, the omfile module has
specifically been written for that use case and, if I remember correctly, the
root cause for its implementation was Apache clf transfer.

It may also be worth noting that in the Apache scenario log4j syslog logging
seems to come together with clf - but I don't have insight if this is true
for the majority of cases. 

On syslog performance: I have read that expected message volume was
considered problematic for the syslog use case. It may be worth noting that
high-volume sites log data via syslog. This may be clf, but the larger ISP or
financial institutions (or other service providers) already have lots of log
data that is to be processed. For rsyslog, I know of deployments that average
50,000+ messages per second on a single receiving machine. In lab setup, a
single instance of rsyslog can currently process up to 250,000 msgs per
second, with this rates going up. The Windows products I am responsible for
reach similar or higher message rates. Of course, these number depend much on
the length of the message, parsing overhead and what the final destination
does with the messages (it is a big difference writing them to a flat ascii
file or a database and complex filtering also reduces the throughput). Note
that there is large demand for even faster syslog implementations, which
leads me to believe that transmission of mass data via syslog is often
desired.

I am not sure what message rates are expected for sip and where the actual
problem for syslog was envisioned. If you have some more information on that,
it would definitely help me understand the situation at large.

Rainer Gerhards