Re: [Json] Using JSON in log files

This is one of canonical use-cases for streaming JSON processing: just use
linefeed between root-level values. Many/most streaming JSON readers
support this, especially Java ones (Jackson definitely does; GSON has
streaming mode, Genson as well; org.json's toy lib should not be used for
anything).
It also makes it easier to read, should you need to check out log files.

Or, if you want to be strictly compliant with specs, use a container JSON
array.
Streaming readers (and generators) should work fine here as well.
There is no need to buffer the whole log stream in buffer either when
reading or writing.

-+ Tatu +-

On Wed, Nov 20, 2013 at 8:09 AM, Phillip Hallam-Baker <hallam@gmail.com>wrote:

> The biggest technical blunder in JSON right now is the insistence that
> there only be one object in a file. XML suffers from the same blunder, here
> is why.
>
> If you have a log file and it is being updated by multiple processes,
> there is potential for contention between the updates. The simplest
> mechanism for resolving that contention is to make the log append only and
> use some operating system primitive that supports an atomic write to the
> end of the file.
>
> I do have code that allows me to append to the end of a JSON log without
> reading the whole thing into memory (which is important as my logs are
> expected to grow to several MB.) What I do is to work back from the end of
> the file until I see "]}" and then I replace that with ", {<New entry>} ]}
>
> But this approach is rather brittle and somewhat risky. It also upsets
> various schemes designed to avoid wearing out my flash memory that handle
> atomic append operations in a special way.
>
>
> What I would like is an option for use in log files only that would allow
> a beginning of array marker to inform the reader that the end of file
> marker is an implicit close for that structure.
>
> So instead of my file starting:
> {
>   "Keys": [{
>       "KeyID": "AAAJMA-ECEYAJ-GABJYPA-G6AAVC-GAB4ACI-FAAFGA-BGZ2AL-LAA",
>       "SelfCertificates": [{
>
> It would look something like:
>
> [*
>   {"Key": [{
>       "KeyID": "AAAJMA-ECEYAJ-GABJYPA-G6AAVC-GAB4ACI-FAAFGA-BGZ2AL-LAA",
>       "SelfCertificates": [{
>    ...}
>   {...}
>   {...}
>
> This is not an extension I would want to use in an on-the-wire format. But
> for log files it is essential. Particularly in my case where the logs are
> digitally signed notary records.
>
> I will be adding a similar feature to JSON-B but I would if possible like
> to make whatever change I make there be compatible with any solutions that
> other people have developed independently.
>
>
> I know this is out of scope for the docs, but I think that adding this
> feature and comments to the text encoding of JSON would be useful recharter
> items.
>
> And yes, I am aware that some people would prefer to keep JSON simple and
> not meet the requirements of my use cases. My position is that I want to
> use a single encoding for all the non ASN.1 parts of my system and that be
> as close to JSON as possible. But I am not going to compromise my system it
> the specification is not adequate for my needs.
>
> Writing a JSON reader is very easy, I am quite OK with a requirement that
> someone has to write their own reader or put a wrapper around an existing
> reader to accept my logs.
>
> --
> Website: http://hallambaker.com/
>
> _______________________________________________
> json mailing list
> json@ietf.org
> https://www.ietf.org/mailman/listinfo/json
>
>