Re: [Syslog] Small draft for Syslog File Storage?
robert.horn@agfa.com Thu, 11 November 2010 22:39 UTC
Return-Path: <prvs=924a79796=robert.horn@agfa.com>
X-Original-To: syslog@core3.amsl.com
Delivered-To: syslog@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6001D3A6358; Thu, 11 Nov 2010 14:39:46 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.598
X-Spam-Level:
X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EM585TQHtHN5; Thu, 11 Nov 2010 14:39:41 -0800 (PST)
Received: from mornm01-out.agfa.com (mornm01-out.agfa.com [134.54.1.75]) by core3.amsl.com (Postfix) with ESMTP id D82663A6A84; Thu, 11 Nov 2010 14:39:28 -0800 (PST)
X-IronPort-AV: E=Sophos;i="4.59,185,1288566000"; d="scan'208";a="114268851"
Received: from morswa037.agfa.be (HELO morswa037.be.local) ([10.232.220.21]) by mornm01-out.agfa.com with ESMTP; 11 Nov 2010 23:39:12 +0100
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6C8@GRFEXC.intern.adiscon.com>
To: rgerhards@hq.adiscon.com
MIME-Version: 1.0
Message-ID: <OF1DA8871B.B17BF5FD-ON852577D7.005E721C-852577D8.007C7242@agfa.com>
From: robert.horn@agfa.com
Date: Thu, 11 Nov 2010 17:39:13 -0500
Content-Type: multipart/alternative; boundary="=_alternative 007C7242852577D8_="
Cc: syslog@ietf.org, syslog-bounces@ietf.org
Subject: Re: [Syslog] Small draft for Syslog File Storage?
X-BeenThere: syslog@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Security Issues in Network Event Logging <syslog.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/syslog>, <mailto:syslog-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/syslog>
List-Post: <mailto:syslog@ietf.org>
List-Help: <mailto:syslog-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/syslog>, <mailto:syslog-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 11 Nov 2010 22:39:46 -0000
A use case that is coming up fairly often in the health care world is the following: 1) System A is gathering syslog reports and archiving them into an internal data structure. The incoming messages comply with the Syslog standard. 2) At some later time, an administrative request is made for "all syslog reports related to Event X". The mechanism for making such a request is not covered by Syslog, but it's a feature of System A. 3) These events are extracted and put into a file format for transfer to System B. This transfer may use media, ftp, or any other method suitable for transfering files. 4) System B analyzes the event reports. A standard format for syslog file storage would cover the file transferred in step 3. Kind Regards, Robert Horn | Agfa HealthCare Research Scientist | HE/Technology Office T +1 978 897 4860 Agfa HealthCare Corporation, Gotham Parkway 580, Carlstadt, NJ 07072-2405, USA http://www.agfa.com/healthcare/ Click on link to read important disclaimer: http://www.agfa.com/healthcare/maildisclaimer "Rainer Gerhards" <rgerhards@hq.adiscon.com> Sent by: syslog-bounces@ietf.org 11/10/2010 02:38 AM To "David Harrington" <ietfdbh@comcast.net>, <syslog@ietf.org> cc Subject Re: [Syslog] Small draft for Syslog File Storage? > -----Original Message----- > From: David Harrington [mailto:ietfdbh@comcast.net] > Sent: Wednesday, November 10, 2010 7:52 AM > To: Rainer Gerhards; syslog@ietf.org > Subject: RE: [Syslog] Small draft for Syslog File Storage? Good questions, as usual. Obviously I have only one voice here, so for the most part, I do not know. Would the OPS area be the right area to ask this in addition to here? My question was motivated by the Mitre CEE effort: http://cee.mitre.org/ In very short words, CEE tries to define a standard event format, where what syslog carries is a subset of the events possible. CEE will also define syntaxes for log storage. We will most probably support XML, CSV, JSON and syslog, with syslog being the only format where only a on-the-wire but no file format standard exists. I am on the CEE board and one thing we currently try to accomplish is define a CEE-to-syslog mapping. There are a couple of the larger vendors interested in logging on the board and the overall consensus seems to be that text files play an important role when it comes to a) storing log messages b) feeding log messages into analysis backends My own experience in the Linux environment and working with larger users confirms that. I have some very large customers (which I cannot name due to NDA) which store logs in (zipped) text file format because any other store is impractical for their needs. Of course, that doesn't exclude representations of other subsets in other formats for other needs. I will try to gather feedback at least from the CEE community, but would appreciate comments from others as well. Rainer > How many syslog sender/receiver implementers would be willing to > support such a common format? > > How many log anaysis application vendors would like such a common > format? or do they consider it unneccesray because they convert > incoming info into their own proprietary database formats anyway? > > dbh > > > -----Original Message----- > > From: syslog-bounces@ietf.org > > [mailto:syslog-bounces@ietf.org] On Behalf Of Rainer Gerhards > > Sent: Wednesday, November 10, 2010 2:24 PM > > To: syslog@ietf.org > > Subject: [Syslog] Small draft for Syslog File Storage? > > > > Hi all, > > > > In what we did, we specified the on-the-wire format. However, > > we did not > > specify any format to use when persisting syslog data to a file. > > > > Note that we were very generous when specifying the > > on-the-wire format, for > > example we permit LF, CR, NUL and many other characters > > considered dangerous > > in file formats. > > > > There are many tools available which interpret syslog data > > stored in text > > files. However, different syslog implementations may use > > slightly different > > file formats. > > > > Together with the control character issue, the file format > > question both has > > interoperability AND security issues. I think these would be > > very easy to fix > > if we write a small RFC that specifies how text is to be > > encoded. It would be > > similar, but much smaller to RFC4627 (JSON). Actually, I > > think we would need > > to carry over primarily its section 2.5. > > > > I would volunteer to write an initial draft, but would first > > like to get some > > feedback if this effort has any chance of getting through. > > > > Rainer > > _______________________________________________ > > Syslog mailing list > > Syslog@ietf.org > > https://www.ietf.org/mailman/listinfo/syslog _______________________________________________ Syslog mailing list Syslog@ietf.org https://www.ietf.org/mailman/listinfo/syslog
- [Syslog] Small draft for Syslog File Storage? Rainer Gerhards
- Re: [Syslog] Small draft for Syslog File Storage? David Harrington
- Re: [Syslog] Small draft for Syslog File Storage? Rainer Gerhards
- Re: [Syslog] Small draft for Syslog File Storage? Simon Josefsson
- Re: [Syslog] Small draft for Syslog File Storage? Rainer Gerhards
- Re: [Syslog] Small draft for Syslog File Storage? Heinbockel, Bill
- Re: [Syslog] Small draft for Syslog File Storage? Simon Josefsson
- Re: [Syslog] Small draft for Syslog File Storage? Chris Lonvick
- Re: [Syslog] Small draft for Syslog File Storage? Rainer Gerhards
- Re: [Syslog] Small draft for Syslog File Storage? Simon Josefsson
- Re: [Syslog] Small draft for Syslog File Storage? Rainer Gerhards
- Re: [Syslog] Small draft for Syslog File Storage? t.petch
- Re: [Syslog] Small draft for Syslog File Storage? Anton Chuvakin
- Re: [Syslog] Small draft for Syslog File Storage? robert.horn