[Json] Using JSON in log files

[Phillip Hallam-Baker <[e-mail]>]

The biggest technical blunder in JSON right now is the insistence that
there only be one object in a file. XML suffers from the same blunder, here
is why.

If you have a log file and it is being updated by multiple processes, there
is potential for contention between the updates. The simplest mechanism for
resolving that contention is to make the log append only and use some
operating system primitive that supports an atomic write to the end of the
file.

I do have code that allows me to append to the end of a JSON log without
reading the whole thing into memory (which is important as my logs are
expected to grow to several MB.) What I do is to work back from the end of
the file until I see "]}" and then I replace that with ", {<New entry>} ]}

But this approach is rather brittle and somewhat risky. It also upsets
various schemes designed to avoid wearing out my flash memory that handle
atomic append operations in a special way.


What I would like is an option for use in log files only that would allow a
beginning of array marker to inform the reader that the end of file marker
is an implicit close for that structure.

So instead of my file starting:
{
  "Keys": [{
      "KeyID": "AAAJMA-ECEYAJ-GABJYPA-G6AAVC-GAB4ACI-FAAFGA-BGZ2AL-LAA",
      "SelfCertificates": [{

It would look something like:

[*
  {"Key": [{
      "KeyID": "AAAJMA-ECEYAJ-GABJYPA-G6AAVC-GAB4ACI-FAAFGA-BGZ2AL-LAA",
      "SelfCertificates": [{
   ...}
  {...}
  {...}

This is not an extension I would want to use in an on-the-wire format. But
for log files it is essential. Particularly in my case where the logs are
digitally signed notary records.

I will be adding a similar feature to JSON-B but I would if possible like
to make whatever change I make there be compatible with any solutions that
other people have developed independently.


I know this is out of scope for the docs, but I think that adding this
feature and comments to the text encoding of JSON would be useful recharter
items.

And yes, I am aware that some people would prefer to keep JSON simple and
not meet the requirements of my use cases. My position is that I want to
use a single encoding for all the non ASN.1 parts of my system and that be
as close to JSON as possible. But I am not going to compromise my system it
the specification is not adequate for my needs.

Writing a JSON reader is very easy, I am quite OK with a requirement that
someone has to write their own reader or put a wrapper around an existing
reader to accept my logs.

-- 
Website: http://hallambaker.com/