Re: [Syslog] Small draft for Syslog File Storage?

Simon Josefsson <simon@josefsson.org> Wed, 10 November 2010 11:52 UTC

Return-Path: <simon@josefsson.org>
X-Original-To: syslog@core3.amsl.com
Delivered-To: syslog@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id EE0D528C0D6 for <syslog@core3.amsl.com>; Wed, 10 Nov 2010 03:52:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.599
X-Spam-Level:
X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RpwINkJgX2An for <syslog@core3.amsl.com>; Wed, 10 Nov 2010 03:52:39 -0800 (PST)
Received: from yxa-v.extundo.com (yxa-v.extundo.com [83.241.177.39]) by core3.amsl.com (Postfix) with ESMTP id D754A3A6AA2 for <syslog@ietf.org>; Wed, 10 Nov 2010 03:52:38 -0800 (PST)
Received: from latte.josefsson.org (c80-216-27-64.bredband.comhem.se [80.216.27.64]) (authenticated bits=0) by yxa-v.extundo.com (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id oAABqVss012806 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 10 Nov 2010 12:52:32 +0100
From: Simon Josefsson <simon@josefsson.org>
To: "Rainer Gerhards" <rgerhards@hq.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com>
OpenPGP: id=B565716F; url=http://josefsson.org/key.txt
X-Hashcash: 1:22:101110:rgerhards@hq.adiscon.com::nxiqM2YuLJP0RdvG:08kh
X-Hashcash: 1:22:101110:syslog@ietf.org::S0h0pLUZxrZPll4T:RqP4
Date: Wed, 10 Nov 2010 12:52:34 +0100
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> (Rainer Gerhards's message of "Wed, 10 Nov 2010 07:24:17 +0100")
Message-ID: <87mxphe6pp.fsf@latte.josefsson.org>
User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: clamav-milter 0.96.3 at yxa-v
X-Virus-Status: Clean
X-Mailman-Approved-At: Wed, 10 Nov 2010 08:04:12 -0800
Cc: syslog@ietf.org
Subject: Re: [Syslog] Small draft for Syslog File Storage?
X-BeenThere: syslog@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Security Issues in Network Event Logging <syslog.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/syslog>, <mailto:syslog-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/syslog>
List-Post: <mailto:syslog@ietf.org>
List-Help: <mailto:syslog-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/syslog>, <mailto:syslog-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Nov 2010 11:52:40 -0000

"Rainer Gerhards" <rgerhards@hq.adiscon.com> writes:

> Hi all,
>
> In what we did, we specified the on-the-wire format. However, we did not
> specify any format to use when persisting syslog data to a file.
>
> Note that we were very generous when specifying the on-the-wire format, for
> example we permit LF, CR, NUL and many other characters considered dangerous
> in file formats.
>
> There are many tools available which interpret syslog data stored in text
> files. However, different syslog implementations may use slightly different
> file formats.
>
> Together with the control character issue, the file format question both has
> interoperability AND security issues. I think these would be very easy to fix
> if we write a small RFC that specifies how text is to be encoded. It would be
> similar, but much smaller to RFC4627 (JSON). Actually, I think we would need
> to carry over primarily its section 2.5.
>
> I would volunteer to write an initial draft, but would first like to get some
> feedback if this effort has any chance of getting through.

I would support that effort.  Multiple incompatible syslog formats are a
pain, and it has bitten me several times in multiple jobs.

One approach that would be easy to move forward with is that you write a
document registering a MIME media sub-type, e.g., text/syslog or
application/syslog.  Describing one reasonable syslog text format in
that document.  With this approach, there is less pressure on all syslog
vendors to agree on a particular format, and you can just invent one
format that makes sense and let people adopt it on an opt-in basis.
I've registered some MIME media types, and learned some details when
doing that, so I could help with this approach if you want.  Having a
MIME media type specified would also enable reliable transfer of syslog
data in protocols that are MIME aware (e.g., HTTP or e-mail).  With this
approach, there is also less of a requirement to be backwards compatible
with existing (often sub-optimal) formats.

/Simon