Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?

[Jeroen Massar <[e-mail]>]

On 2011-02-16 06:21, Heinbockel, Bill wrote:
> From what I understand, IPFIX is for expression of IP flows from network sensing
> devices.

For a short bit forget about the history of IPFIX, it indeed comes from
NetFlow, and thus is used quite in a network centric way, but
effectively it is a structured streaming data format.

> Could you please explain how IPFIX is relevant to event and cloud logging data?
> I understand how CEE and IPFIX may overlap for describing networking events, but
> it is unclear to me how IPFIX could handle things like Windows Event Logs and
> RHEL audit logs.

There are two parts to IPFIX: Templates + Data

The template describes how the data looks like, for instance, lets take
an Apache CLF log entry:

66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt
HTTP/1.1" 200 2629 "-" "Googlebot-Image/0"

We can make an IPFIX template for that

[
	{4, IPv4_SRC },
	{4, TIMESTAMP},
	{4, HTTP_METHOD},
	{v, URL},
	{v, HTTP_PROTOCOL},
	{2, HTTP_RESULT},
	{8, OCTETS},
	{v, HTTP_REFER},
	{v, HTTP_USERAGENT},
]

The 'v' markers indicate variable fieldlengths, the others indicates the
number of bytes such a field takes. The data is then just encoded in the
above format, presto.

The above is a simple example, one can also have repeating lists and of
course you could make a variable template which just includes the fields
that you actually want to look at or you could already do some
aggregation and add other fields. Templates are only sent every now and
then, as they should not change. The data is the important bit.

The fieldnames are actually numbers in the data, thus very compact, and
are mapped to descriptions, data types etc, per a nice XML file
 http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt for
a more human readable version ;) for the official IANA list and with the
help of Enterprise IDs any others can easily be added.

The big advantage is that you can more or less do static templates if
you want and you only need one single parser on the collector side, thus
one does not have to create another parser and collector again for
decoding other protocols, just one, the IPFIX one, and you can optimize
that really well for all kinds of scenarios.

Greets,
 Jeroen