Re: [Json] Using JSON in log files

Nico Williams <nico@cryptonector.com> Thu, 21 November 2013 02:21 UTC

Return-Path: <nico@cryptonector.com>
X-Original-To: json@ietfa.amsl.com
Delivered-To: json@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4AAD31ADF99 for <json@ietfa.amsl.com>; Wed, 20 Nov 2013 18:21:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.378
X-Spam-Level:
X-Spam-Status: No, score=-1.378 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tr8GTHlQGkA6 for <json@ietfa.amsl.com>; Wed, 20 Nov 2013 18:21:56 -0800 (PST)
Received: from homiemail-a88.g.dreamhost.com (mailbigip.dreamhost.com [208.97.132.5]) by ietfa.amsl.com (Postfix) with ESMTP id 6DA0A1ADF87 for <json@ietf.org>; Wed, 20 Nov 2013 18:21:56 -0800 (PST)
Received: from homiemail-a88.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a88.g.dreamhost.com (Postfix) with ESMTP id F0269264059 for <json@ietf.org>; Wed, 20 Nov 2013 18:21:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cryptonector.com; h= mime-version:in-reply-to:references:date:message-id:subject:from :to:cc:content-type; s=cryptonector.com; bh=KTNqbubbSIMeVS9R70NO WyR5Bxo=; b=MorQGvocVfRuABhtVW9fBmxuIACWDk7Dmhw1jtIPMm0Sfjx1cZds iW8n/xG2bJ2G1tiEM6IgJt1XR/oCxlcGFKTSk7reFXrM4/Vk0f/b2++bnLGNd04w gXVIcJQyjs6HRA5vXifuyo+95XYfi8Qq7b20r9ZrN+93uJJK+SkqQ0c=
Received: from mail-we0-f171.google.com (mail-we0-f171.google.com [74.125.82.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: nico@cryptonector.com) by homiemail-a88.g.dreamhost.com (Postfix) with ESMTPSA id 99E15264005 for <json@ietf.org>; Wed, 20 Nov 2013 18:21:49 -0800 (PST)
Received: by mail-we0-f171.google.com with SMTP id q58so4200290wes.2 for <json@ietf.org>; Wed, 20 Nov 2013 18:21:47 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=htag79+zxigDw0ubYZ4qhpXVpF2esXRLUvrIkO1+wtk=; b=DhBEDl5vFr28yMF5Lto9/c9QM5M85mC9K/pN2C7ArCJrmIE14dtPhgT9yGvkjXUneu tJkOBbHrRrKkiViakFKIOkutS1ftHOSVt+CWp358sVPTf2dj1hrtLAaR/hd9vEWIbULO Agb6atOcado1hw1AZMtGRo2vcd9cAOLEmfkVaXRkYD6mtsUOHH3efGnjadjVFNtFsOh6 eOI/N6S93MS8YYtpH2t1H2zLcmIu5k56xp4vWyT2LlrnjzeYC30XP+BqReWgpzpYc2Fg TpMiZSVlpPOHekBhQc7JPNScZxVw8SwYnVdnUJUtk3t9m8mc+YgSIHNApv0SAdoGnS2B v8Mg==
MIME-Version: 1.0
X-Received: by 10.180.39.212 with SMTP id r20mr27954779wik.13.1385000507786; Wed, 20 Nov 2013 18:21:47 -0800 (PST)
Received: by 10.216.151.136 with HTTP; Wed, 20 Nov 2013 18:21:47 -0800 (PST)
In-Reply-To: <CAMm+Lwj49w5qq3V8tLta_GPq3TT5A0FXuKww5RXHbe74dQ5jjA@mail.gmail.com>
References: <CAMm+Lwj49w5qq3V8tLta_GPq3TT5A0FXuKww5RXHbe74dQ5jjA@mail.gmail.com>
Date: Wed, 20 Nov 2013 20:21:47 -0600
Message-ID: <CAK3OfOi+s7syMF=Eq+GaqFPKxjqK1fkMsLvv+YbduwRTzg=EhQ@mail.gmail.com>
From: Nico Williams <nico@cryptonector.com>
To: Phillip Hallam-Baker <hallam@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Cc: JSON WG <json@ietf.org>
Subject: Re: [Json] Using JSON in log files
X-BeenThere: json@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "JavaScript Object Notation \(JSON\) WG mailing list" <json.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/json>, <mailto:json-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/json/>
List-Post: <mailto:json@ietf.org>
List-Help: <mailto:json-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/json>, <mailto:json-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Nov 2013 02:21:57 -0000

On Wed, Nov 20, 2013 at 10:09 AM, Phillip Hallam-Baker <hallam@gmail.com> wrote:

+1

If you look at tools like jq, you'll see that it's perfectly natural
to treat a sequence of JSON texts as an array of values, without any
need to first collect them into a larger JSON text consisting of an
array of all those values.

% echo 1 2 3 4 | jq .
1
2
3
4
% echo 1 2 3 4 | jq -s .
[
  1,
  2,
  3,
  4
]
%

This approach, combined with an online parser, makes it trivial to
deal with log-type structures (append-only with atomic appends) where
one appends JSON texts.

Sure, one could also pretend that the log starts with '[', make sure
every entry is a JSON text followed by a ',', and use an online
parser, but this is more fun.

The one caveat is that numbers are not self-delimiting, which means
that JSON texts consisting of a numeric value must be delimited with
whitespace.

Nico
-- 

Nico
--