[sip-clf] A primer on syslog.

["David Harrington" <[e-mail]>]

Hi,

For those unacquainted with syslog, let me try to summarize how it
works. I will focus on the forwarding and filtering and security
mechanisms, since I think those will be important considerations for
SIP logging.

Rainer is the guy I go to when I have a question about syslog. He has
answered Vijay's concern in considerable detail in another email. I
don't personally deploy syslog, so Rainer can feel free to correct
anything I get wrong. ;-)

I will recommend looking at the wikipedia entry, which has a summary
of its history and usage, but some factual details about the protocol
are about the limitations of BSD syslog, rather than the IETF standard
syslog. As technical advisor, I recommend the IETF standard syslog
because it overcomes many of the limitations and interoperability
issues with BSD syslog. The wikipedia entry mentions some work that is
being done, work that relates to anomaly detection and security
management systems.

The IETF has worked at standardizing syslog across operating systems,
and focused mainly on standardizing aspects of syslog that impacted
security and filtering and correlation (such as standardizing the
header, and more detailed info about the source and time of the
messages).

---- The forwarding mechanism ----

Syslog allows log information for both kernel processes and user
processes to be forwarded to a file, a set of users, or another node
in a network. 

IETF syslog provides support for UDP, TLS, and (soon) DTLS. It is
designed to allow additional transports to be defined, so could also
be used in non-IP environments, I suppose. 

This flexibility in the forwarding rules allows syslog to concentrate
on collecting data and putting it where the operator wants it so other
applications can focus on sorting, filtering, and alerting on the
contents. Forwarding to a collector elsewhere in the network can help
prevent attackers from modifying the local logs to cover their tracks,
and allows a centralized location for operators and tools to do log
analysis rather than having to go to each device to look at the logs.
But you can also store it locally if you want.

---- The filtering mechanism ----

syslog messages start with a <priority> byte that is calculated from
two factors - the facility that generated the message and the severity
of the message. This is the primary mechanism for sorting messages
that are logged.

Facilities define the source of a syslog message. There are 23
different facilities defined for use in the <priority field>, ranging
from "kernel" to system daemons to mail to security to local
applications. A SIP application would use one of the "local"
facilities to log messages. If there are multiple SIP applications,
they might use different "local" facilities to generate separate
logging streams for forwarding purposes, but whether this is done
depends on the implementations and operator configurations. Additional
header fields and the SDE feature of IETF syslog allow much finer
filtering during log analysis.

There are 7 levels of severity ranging from emergency to critical to
informational to debug messages. The IETF standard syslog severities
can be aligned with the ITU-T values of perceived severity (M.2100,
X.733, X.736), and with the SNMP values defined in the ALARM-MIB. This
helps operators correlate events reported by different applications or
by different mechanisms.

syslog.conf is the normal mechanism for configuring what to log and
how to distribute it. The filtering/forwarding instructions look
something like this:

mail.info	/var/log/maillog	-- send all informational
mail-related messages
                                 to the file /var/log/maillog
cron.*      /var/log/cron     -- send all messages from cron,
regardless of
                                 severity to the file /var/log/cron
security.*  @loghost.netlab.lander.edu
                              -- send all security related messages,
regardless
                                 of severity to a remote system
(loghost.netlab.lander.edu)
*.notice     root             -- send all notices, regardless of
source, to 
                                 the root user, if root is logged on.

and so on.

These are all configured administratively. Operators are usually very
acquainted with syslog as a tool. 

---- Post logging analysis ----

Other header values can also be used for analyzing logs, including
timestamp, hostname, application name, process ID, etc. Filtering
based on header values has been in wide use for a long time, although
the header format for different syslog implementations vary widely.
IETF syslog standardizes the order and format of header fields, but
still keeps roughly the same historic header fields. There are many
tools available for syslog log analysis.

The Structured Data Elements provide even more capability. This is
only available in the IETF standard syslog. My earlier message pointed
out some of the already-standardized SDEs defined by the WG that can
be used to filter log entries.

Hope this helps,
dbh

> -----Original Message-----
> From: Spencer Dawkins [mailto:[e-mail]] 
> Sent: Wednesday, February 03, 2010 11:46 AM
> To: David Harrington; 'Vijay K. Gurbani'; [e-mail]
> Subject: Re: [sip-clf] WGLC: SIPCLF 
> ProblemStatement(draft-gurbani-sipclf-problem-statement-01)
> 
> Hi, Dave,
> 
> <as-co-chair>
> 
> Thanks for clarifying your role - it's helpful.
> 
> </as-co-chair>
> 
> <as-participant>
> 
> On one specific point:
> >
> > As I read the problem-statement-01, my
> > impression is syslog and/or ipfix might fit well to meet certain
> > goals, however, it is highly possible that syslog and ipfix 
> are WRONG
> > to solve the sipclf problem.
> >
> > As Technical Advisor - not as technical contributor - I 
> recommend that
> > the WG consider syslog and ipfix seriously. My advice as Technical
> > Advisor is to do a real analysis about the applicability of
existing
> > standards like syslog and ipfix to the various sip clf use 
> cases. This
> > should not be just a cursory analysis; we spent probably 
> less than one
> > minute talking about syslog at the last IETF meeting.
> 
> I think the discussion lasted longer than a minute, but I 
> suspect the longer 
> time I'm remembering included people walking to the mike :D
> 
> So, my understanding of the pushback on SYSLOG in Hiroshima - 
> and I think 
> this is what Vijay was saying in his followup - was that 
> SYSLOG entries 
> become really important when Bad Things Happen(TM), and 
> people weren't sure 
> that producing a lot of SIPCLF SYSLOG entries, which might even be 
> message-by-message for a large SIP proxy, that you needed to 
> prune through 
> in order to find more "interesting" SYSLOG entries like "file 
> system full", 
> was the right thing to do.
> 
> Since you are a smart guy about SYSLOG - is this a realistic 
> concern for us 
> to have? and, if the answer is "yes", does modern SYSLOG with 
> structured 
> data elements provide a way to either (1) quickly separate 
> the SIPCLF SYSLOG 
> entries from other "oh, no" SYSLOG entries, or (2) send 
> SIPCLF log entries 
> using the same protocol to two different destinations, one 
> potentially 
> containing massive amounts of SIPCLF log entries, and the 
> other containing 
> "everything else"?
> 
> I would really like to use one of the existing management 
> frameworks (still 
> speaking as a participant here), IFF it's the right thing to do.
> 
> </as-participant>
> 
> Thanks,
> 
> Spencer 
> 
>