Re: [Syslog] Small draft for Syslog File Storage?

"Heinbockel, Bill" <> Wed, 10 November 2010 19:28 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 275A23A6887 for <>; Wed, 10 Nov 2010 11:28:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -5.949
X-Spam-Status: No, score=-5.949 tagged_above=-999 required=5 tests=[AWL=0.650, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id jZmaPHpeV-o9 for <>; Wed, 10 Nov 2010 11:28:34 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id C699B3A6821 for <>; Wed, 10 Nov 2010 11:28:33 -0800 (PST)
Received: from (localhost.localdomain []) by (8.13.1/8.13.1) with ESMTP id oAAJT1T2025539 for <>; Wed, 10 Nov 2010 14:29:01 -0500
Received: from imchub2.MITRE.ORG ( []) by (8.13.1/8.13.1) with ESMTP id oAAJT1xS025534 for <>; Wed, 10 Nov 2010 14:29:01 -0500
Received: from IMCMBX3.MITRE.ORG ([]) by imchub2.MITRE.ORG ([]) with mapi; Wed, 10 Nov 2010 14:29:01 -0500
From: "Heinbockel, Bill" <>
To: "" <>
Date: Wed, 10 Nov 2010 14:26:13 -0500
Thread-Topic: Re: [Syslog] Small draft for Syslog File Storage?
Thread-Index: AcuBDSGslkXMwmgqScKcXKhlp6ijLw==
Message-ID: <93ED0A84F9A1D74FA65021D940AA5884053E827EBF@IMCMBX3.MITRE.ORG>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_0005_01CB80E3.38DFBE70"
MIME-Version: 1.0
Subject: Re: [Syslog] Small draft for Syslog File Storage?
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Security Issues in Network Event Logging <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 10 Nov 2010 19:29:20 -0000

Sounds like a good idea to me

The biggest step that you need to make from the on-the-wire RFC5424 Syslog is
the specification of a Syslog record separator. In most Syslog log files (as
well as CSV and other multi-record file formats), the typical record separator
is LF or CRLF.
Regardless, in order to define the record separator, you will have to add at
least one more encoding or Syslog syntax requirement on top of the existing
RFC5424 specification, as currently all characters are valid in a Syslog message

The specification would be fairly straight-forward, as you could just
standardize on the approaches taken by rsyslog and Syslog-ng. Also, RFC5424
provides enough flexibility in character escaping to build on further escaping
for control characters (U+0000 through U+001F) to make this a possibility

In addition, I would like to suggest the addition of an optional file header for
Syslog files. This would allow for easy versioning of the file, allow a place
for products to include additional information, and be able to hold information
such as the vendor, name, and version of the application producing the log. This
would be an especially nice feature when digging through and parsing old Syslog

Regardless of the outcome of this discussion, I would like to see a couple of
more optional encodings added to the RFC5424 specification to handle U+0000
through U+001F characters
maybe: \n, \r, \t, and some generic hex encoding for the others \x00 \x01 ...

> -----Original Message-----
> From: syslog-bounces at 
> [mailto:syslog-bounces at] On Behalf Of Rainer Gerhards
> Sent: Wednesday, November 10, 2010 2:24 PM
> To: syslog at
> Subject: [Syslog] Small draft for Syslog File Storage?
> Hi all,
> In what we did, we specified the on-the-wire format. However, 
> we did not
> specify any format to use when persisting syslog data to a file.
> Note that we were very generous when specifying the 
> on-the-wire format, for
> example we permit LF, CR, NUL and many other characters 
> considered dangerous
> in file formats.
> There are many tools available which interpret syslog data 
> stored in text
> files. However, different syslog implementations may use 
> slightly different
> file formats.
> Together with the control character issue, the file format 
> question both has
> interoperability AND security issues. I think these would be 
> very easy to fix
> if we write a small RFC that specifies how text is to be 
> encoded. It would be
> similar, but much smaller to RFC4627 (JSON). Actually, I 
> think we would need
> to carry over primarily its section 2.5.
> I would volunteer to write an initial draft, but would first 
> like to get some
> feedback if this effort has any chance of getting through.
> Rainer
> _______________________________________________
> Syslog mailing list
> Syslog at