[Json] JSON text sequences and semi-online parsing (Re: Regarding JSON text sequence ambiguities (...)

[Nico Williams <[e-mail]>]

On Thu, Mar 13, 2014 at 4:43 PM, Tim Bray <[e-mail]> wrote:
> I’m halfway convinced that it might be useful to say that if you are
> persisting something as chunks of JSON, use X as a separator.  I wouldn’t be

But they aren't chunks.  They are proper, fully-formed JSON texts.

> surprised to see the advent of standardized log-as-JSON libraries. I’d go

Neither would I.

> further and say that if you are doing this, use objects too.  Having said

I'm going to write a separate reply to this.  Your insistence on
top-level objectness deserves a complete treatment.

> that, it’s not obvious that this is the kind of format that you’d
> interchange much, so the case for a media type is weak.

Let's talk about not-online, online, and what I'll call semi-online parsing.

Consider jq once more.  At this time jq's parser handles incremental
parsing, but is not online.  This means that it can parse a top-level
array of 10e6 values, say, incrementally but it won't produce any
values until the final ']' is read.  But jq gladly reads JSON text
sequences, producing each top-level value as it is read.  Therefore jq
can handle JSON text sequences in a semi-online way: online for the
sequence, not online for each element.

A semi-online parsing approach actually happens to be a very useful
way to deal with logs and databases, where you know you're dealing
with a sequence of things, possibly of indeterminate size, possibly
infinite (tail -f ...), but where each element of the sequence is
generally expected to be small.

Not-at-all-online parsing simply does not work in these cases.  But
fully-online parsing is a real PITA, since the parser then produces
outputs like [<path>, <leaf value>] or worse, and one is likely to
aggregate results that form discrete entities, usually one or two
levels down from the top-level.

Now, it should be possible to build a semi-online JSON parser that is
online only for top-level arrays.  But if you have a not-online JSON
parser you can just use JSON text sequences to build a semi-online
JSON text sequence parser.

I.e., it's trivial to implement semi-online parsing of JSON text
sequences with any off-the-shelf JSON text parser.

That's *huge* win because not-online JSON parsers are a dime a dozen,
and a semi-online parse is pretty much the sweet spot for online vs.
simple.  This is exactly what jq does, as described above.

Nico
--