Re: [sip-clf] A syslog approach to sip logging

["Spencer Dawkins" <[e-mail]>]

Rainer,

<as-co-chair>

Welcome, and thanks for subscribing!

</as-co-chair>

<as-participant>

Your note is very interesting to me - I just fired off a question to Dave 
on-list that might also be helpful for you to think about. I'd appreciate 
any thoughts you can provide!

</as-participant>

Thanks,

Spencer


> Hi all,
>
> I just subscribed to the sip-clf mailing list. I am the author of rsyslog,
> one of the major open source syslogd's as well as the designer for a 
> number
> of Windows tools that are syslog-based. I have also worked on the IETF 
> syslog
> standardization effort.
>
> David has made me aware of the current discussion and I am currently 
> working
> through the mailing list. Two things that I would like to comment on are
> transmission of Apache logs via syslog and syslog performance.
>
> As of my experience, it is quite common to transport Apache clf "files" 
> via
> syslog. There are two was to do this: one is to make apache log in 
> real-time
> to the syslogd, usually with the help of logger or a similar system tool.
> This requires proper engineering and can potentially cause notable
> performance degradation. As I know from the rsyslog user base, these 
> problems
> can be solved and this mode is used in practice, even for high-performance
> sites.
>
> The other approach is to let apache write to text files and then transfer
> these text files in near-realtime to a syslogd. That is, a process grabs 
> data
> as it is appended to the text log. In rsyslog, the omfile module has
> specifically been written for that use case and, if I remember correctly, 
> the
> root cause for its implementation was Apache clf transfer.
>
> It may also be worth noting that in the Apache scenario log4j syslog 
> logging
> seems to come together with clf - but I don't have insight if this is true
> for the majority of cases.
>
> On syslog performance: I have read that expected message volume was
> considered problematic for the syslog use case. It may be worth noting 
> that
> high-volume sites log data via syslog. This may be clf, but the larger ISP 
> or
> financial institutions (or other service providers) already have lots of 
> log
> data that is to be processed. For rsyslog, I know of deployments that 
> average
> 50,000+ messages per second on a single receiving machine. In lab setup, a
> single instance of rsyslog can currently process up to 250,000 msgs per
> second, with this rates going up. The Windows products I am responsible 
> for
> reach similar or higher message rates. Of course, these number depend much 
> on
> the length of the message, parsing overhead and what the final destination
> does with the messages (it is a big difference writing them to a flat 
> ascii
> file or a database and complex filtering also reduces the throughput). 
> Note
> that there is large demand for even faster syslog implementations, which
> leads me to believe that transmission of mass data via syslog is often
> desired.
>
> I am not sure what message rates are expected for sip and where the actual
> problem for syslog was envisioned. If you have some more information on 
> that,
> it would definitely help me understand the situation at large.
>
> Rainer Gerhards
> _______________________________________________
> sip-clf mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sip-clf