[Json] Using a non-whitespace separator (Re: Working Group Last Call on draft-ietf-json-text-sequence)

[Nico Williams <[e-mail]>]

I actually like the idea, but I'm concerned about backwards
compatibility for jq.  Help me reason through this.  Help sell me on
this.

Currently my thinking is that for backwards compatibility reasons I'd
want to to make this RECOMMENDED though, not REQUIRED, except for
cases where incomplete writes are a potential problem.  In jq this
would be an option to either use or maybe not use this new separator.

Another option is to say that encoders MUST use the new separator, but
parsers MAY/SHOULD/MUST handle sequences with a missing separator (as
jq does; see below).  jq would still have an encoding option, but when
not emitting the new separator the result just wouldn't be a JSON text
sequence.

FWIW, this is what the jq processor does to handle sequences: it reads
input bytes, feeds them to its parser (which works incrementally, but
isn't streaming), and passes each parsed output to the jq VM to use as
an input to the jq program.  Output values of the jq program are
encoded as JSON texts, printed, and then a newline is printed.

The jq processor has no special handling of newlines on input.  If
there's any bytes left over from parsing a previous text, they are
used in the next parse.  Whitespace is just whitespace.

The only special thing that the jq processor does is to print a
newline after each text on output.

This means that jq can handle JSON text sequences with any whitespace
separator, and even no separator when there would be no ambiguity:

% /jq -c .<<EOF
1 2 true false null"a string""another"[0,1,2
]{"foo":"bar"}
EOF
1
2
true
false
null
"a string"
"another"
[0,1,2]
{"foo":"bar"}
%

I could teach jq how to parse a non-whitespace control character
separator; that's easy enough.  The question is: how to handle
backwards compatibility?  The obvious answer is: add an option.  But
which way should it default?

Nico
--