Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?

Dan Schlitt <schlitt@world.std.com> Thu, 17 February 2011 17:20 UTC

Return-Path: <schlitt@world.std.com>
X-Original-To: syslog@core3.amsl.com
Delivered-To: syslog@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 412ED3A6D1C for <syslog@core3.amsl.com>; Thu, 17 Feb 2011 09:20:47 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.599
X-Spam-Level:
X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nIykDw-v6GUy for <syslog@core3.amsl.com>; Thu, 17 Feb 2011 09:20:45 -0800 (PST)
Received: from TheWorld.com (pcls4.std.com [192.74.137.84]) by core3.amsl.com (Postfix) with ESMTP id 9A04F3A6CB8 for <syslog@ietf.org>; Thu, 17 Feb 2011 09:20:45 -0800 (PST)
Received: from shell.TheWorld.com (root@shell01.TheWorld.com [192.74.137.71]) by TheWorld.com (8.13.6/8.13.6) with ESMTP id p1HHL3M4031579; Thu, 17 Feb 2011 12:21:05 -0500
Received: from shell01.TheWorld.com (localhost.theworld.com [127.0.0.1]) by shell.TheWorld.com (8.13.6/8.12.8) with ESMTP id p1HHL2LK5838276; Thu, 17 Feb 2011 12:21:02 -0500 (EST)
Received: from localhost (schlitt@localhost) by shell01.TheWorld.com (8.13.6/8.13.6/Submit) with ESMTP id p1HHKxsS5828902; Thu, 17 Feb 2011 12:20:59 -0500 (EST)
X-Authentication-Warning: shell01.TheWorld.com: schlitt owned process doing -bs
Date: Thu, 17 Feb 2011 12:20:58 -0500
From: Dan Schlitt <schlitt@world.std.com>
X-X-Sender: schlitt@shell01.TheWorld.com
To: "Heinbockel, Bill" <heinbockel@mitre.org>
In-Reply-To: <93ED0A84F9A1D74FA65021D940AA5884054477F158@IMCMBX3.MITRE.ORG>
Message-ID: <Pine.SGI.4.61.1102171213350.5801294@shell01.TheWorld.com>
References: <4D5A60C8.3090000@unfix.org> <93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> <93ED0A84F9A1D74FA65021D940AA5884054477F158@IMCMBX3.MITRE.ORG>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: Sam Johnston <sj@google.com>, Event Expression <cee@mitre.org>, "syslog@ietf.org" <syslog@ietf.org>, Common@core3.amsl.com
Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?
X-BeenThere: syslog@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Security Issues in Network Event Logging <syslog.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/syslog>, <mailto:syslog-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/syslog>
List-Post: <mailto:syslog@ietf.org>
List-Help: <mailto:syslog-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/syslog>, <mailto:syslog-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 17 Feb 2011 17:20:47 -0000

On this topic it might be well to look at Tom Limoncelli's article in 
the February Communications of the ACM. He is highly respected in the 
system administrator community and speaks for many of us.

/dan

-- 

Dan Schlitt
schlitt@world.std.com


On Thu, 17 Feb 2011, Heinbockel, Bill wrote:

> Thanks for that, I did not realize that IPFIX had been extended beyond its netflow
> past.
>
> I don't have the time now, but I am interested in looking into it further. It does
> kind of remind me of ASN.1/SNMP, where we need to worry about the names/OID
> translation
>
> Also, a lot of vendors and users seem to prefer the ease of text-based protocols
> like Syslog for logging. I am not saying this is good or bad, but it seems to be
> the sweetspot -- binary is not natively human readable and XML has too much
> overhead.
>
>
> William Heinbockel
> The MITRE Corporation
>
>
>> -----Original Message-----
>> From: Jeroen Massar [mailto:jeroen@unfix.org]
>> Sent: Wednesday, 16 February 2011 05:35
>> To: Heinbockel, Bill
>> Cc: syslog@ietf.org; Chris Lonvick; Gene Golovinsky; Sam Johnston; Common
>> Event Expression
>> Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?
>>
>> On 2011-02-16 06:21, Heinbockel, Bill wrote:
>>> From what I understand, IPFIX is for expression of IP flows from network
>> sensing
>>> devices.
>>
>> For a short bit forget about the history of IPFIX, it indeed comes from
>> NetFlow, and thus is used quite in a network centric way, but
>> effectively it is a structured streaming data format.
>>
>>> Could you please explain how IPFIX is relevant to event and cloud logging
>> data?
>>> I understand how CEE and IPFIX may overlap for describing networking
>> events, but
>>> it is unclear to me how IPFIX could handle things like Windows Event Logs
>> and
>>> RHEL audit logs.
>>
>> There are two parts to IPFIX: Templates + Data
>>
>> The template describes how the data looks like, for instance, lets take
>> an Apache CLF log entry:
>>
>> 66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt
>> HTTP/1.1" 200 2629 "-" "Googlebot-Image/0"
>>
>> We can make an IPFIX template for that
>>
>> [
>> 	{4, IPv4_SRC },
>> 	{4, TIMESTAMP},
>> 	{4, HTTP_METHOD},
>> 	{v, URL},
>> 	{v, HTTP_PROTOCOL},
>> 	{2, HTTP_RESULT},
>> 	{8, OCTETS},
>> 	{v, HTTP_REFER},
>> 	{v, HTTP_USERAGENT},
>> ]
>>
>> The 'v' markers indicate variable fieldlengths, the others indicates the
>> number of bytes such a field takes. The data is then just encoded in the
>> above format, presto.
>>
>> The above is a simple example, one can also have repeating lists and of
>> course you could make a variable template which just includes the fields
>> that you actually want to look at or you could already do some
>> aggregation and add other fields. Templates are only sent every now and
>> then, as they should not change. The data is the important bit.
>>
>> The fieldnames are actually numbers in the data, thus very compact, and
>> are mapped to descriptions, data types etc, per a nice XML file
>> http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt for
>> a more human readable version ;) for the official IANA list and with the
>> help of Enterprise IDs any others can easily be added.
>>
>> The big advantage is that you can more or less do static templates if
>> you want and you only need one single parser on the collector side, thus
>> one does not have to create another parser and collector again for
>> decoding other protocols, just one, the IPFIX one, and you can optimize
>> that really well for all kinds of scenarios.
>>
>> Greets,
>> Jeroen
>