Re: [sip-clf] A syslog approach to sip logging

"Spencer Dawkins" <spencer@wonderhamster.org> Wed, 03 February 2010 16:49 UTC

Return-Path: <spencer@wonderhamster.org>
X-Original-To: sip-clf@core3.amsl.com
Delivered-To: sip-clf@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 05CF23A6962 for <sip-clf@core3.amsl.com>; Wed, 3 Feb 2010 08:49:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.176
X-Spam-Level:
X-Spam-Status: No, score=-2.176 tagged_above=-999 required=5 tests=[AWL=-0.178, BAYES_00=-2.599, J_CHICKENPOX_31=0.6, STOX_REPLY_TYPE=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NrPd2K-8usLz for <sip-clf@core3.amsl.com>; Wed, 3 Feb 2010 08:49:56 -0800 (PST)
Received: from mout.perfora.net (mout.perfora.net [74.208.4.194]) by core3.amsl.com (Postfix) with ESMTP id 10A493A68E7 for <sip-clf@ietf.org>; Wed, 3 Feb 2010 08:49:56 -0800 (PST)
Received: from S73602b (cpe-76-182-230-135.tx.res.rr.com [76.182.230.135]) by mrelay.perfora.net (node=mrus1) with ESMTP (Nemesis) id 0M5MVr-1NxpqE2xvr-00z82B; Wed, 03 Feb 2010 11:50:00 -0500
Message-ID: <975D21298CD4428ABEA4FC9D13E7D48C@china.huawei.com>
From: "Spencer Dawkins" <spencer@wonderhamster.org>
To: "Rainer Gerhards" <rgerhards@hq.adiscon.com>, <sip-clf@ietf.org>
References: <9B6E2A8877C38245BFB15CC491A11DA71037F7@GRFEXC.intern.adiscon.com>
Date: Wed, 3 Feb 2010 10:49:44 -0600
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5843
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.5579
X-Provags-ID: V01U2FsdGVkX1+2ZYlJDD1mW7vNGIXbnsCTKBiRMlLF5bkSiWo yUTBt2NLeYgfnav+dSsWZnOcXE0AlyNbi+EogGETjWxhYAEsuD NbSxsx6rGb7vg1cZIwnCMKb0nGKUm32ylYtinUjdsw=
Subject: Re: [sip-clf] A syslog approach to sip logging
X-BeenThere: sip-clf@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: SIP Common Log File format discussion list <sip-clf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/sip-clf>, <mailto:sip-clf-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/sip-clf>
List-Post: <mailto:sip-clf@ietf.org>
List-Help: <mailto:sip-clf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sip-clf>, <mailto:sip-clf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 03 Feb 2010 16:49:57 -0000

Rainer,

<as-co-chair>

Welcome, and thanks for subscribing!

</as-co-chair>

<as-participant>

Your note is very interesting to me - I just fired off a question to Dave 
on-list that might also be helpful for you to think about. I'd appreciate 
any thoughts you can provide!

</as-participant>

Thanks,

Spencer


> Hi all,
>
> I just subscribed to the sip-clf mailing list. I am the author of rsyslog,
> one of the major open source syslogd's as well as the designer for a 
> number
> of Windows tools that are syslog-based. I have also worked on the IETF 
> syslog
> standardization effort.
>
> David has made me aware of the current discussion and I am currently 
> working
> through the mailing list. Two things that I would like to comment on are
> transmission of Apache logs via syslog and syslog performance.
>
> As of my experience, it is quite common to transport Apache clf "files" 
> via
> syslog. There are two was to do this: one is to make apache log in 
> real-time
> to the syslogd, usually with the help of logger or a similar system tool.
> This requires proper engineering and can potentially cause notable
> performance degradation. As I know from the rsyslog user base, these 
> problems
> can be solved and this mode is used in practice, even for high-performance
> sites.
>
> The other approach is to let apache write to text files and then transfer
> these text files in near-realtime to a syslogd. That is, a process grabs 
> data
> as it is appended to the text log. In rsyslog, the omfile module has
> specifically been written for that use case and, if I remember correctly, 
> the
> root cause for its implementation was Apache clf transfer.
>
> It may also be worth noting that in the Apache scenario log4j syslog 
> logging
> seems to come together with clf - but I don't have insight if this is true
> for the majority of cases.
>
> On syslog performance: I have read that expected message volume was
> considered problematic for the syslog use case. It may be worth noting 
> that
> high-volume sites log data via syslog. This may be clf, but the larger ISP 
> or
> financial institutions (or other service providers) already have lots of 
> log
> data that is to be processed. For rsyslog, I know of deployments that 
> average
> 50,000+ messages per second on a single receiving machine. In lab setup, a
> single instance of rsyslog can currently process up to 250,000 msgs per
> second, with this rates going up. The Windows products I am responsible 
> for
> reach similar or higher message rates. Of course, these number depend much 
> on
> the length of the message, parsing overhead and what the final destination
> does with the messages (it is a big difference writing them to a flat 
> ascii
> file or a database and complex filtering also reduces the throughput). 
> Note
> that there is large demand for even faster syslog implementations, which
> leads me to believe that transmission of mass data via syslog is often
> desired.
>
> I am not sure what message rates are expected for sip and where the actual
> problem for syslog was envisioned. If you have some more information on 
> that,
> it would definitely help me understand the situation at large.
>
> Rainer Gerhards
> _______________________________________________
> sip-clf mailing list
> sip-clf@ietf.org
> https://www.ietf.org/mailman/listinfo/sip-clf