Re: [Json] Using a non-whitespace separator (Re: Working Group Last Call on draft-ietf-json-text-sequence)

[Nico Williams <[e-mail]>]

On Thursday, June 5, 2014, Manger, James <[e-mail]>
wrote:

> >> JSON-sequence = *( ws %1e JSON-text )
>
> RS as a JSON sequence prefix or separator was a bad idea when discussed a
> month ago and still is.


I'm happy to go with the current I-D's LF-based recovery mechanism or
variant of it.  If that's not acceptable then something like RS has to be
it.

A third alternative is to abandon logfiles as a use or accept truncated
writes leading to fatal parse errors from the corrupted point forwards.  A
fourth alternative is to not publish on the Standards track, maybe publish
as Informational or Experimental whatever fewer of us think is best.
 Before we get to any of those alternatives I'd like to try to get
consensus.

Remember, it's rough consensus and running code.  I have running code and
I'm open to changing it, but if some views are mutually exclusive and
therefore some have to be on the rough side of consensus, then so be it.


> * You cannot (easily) enter an RS in notepad.


A reason to make it optional, as I proposed -- only logfile writers should
have to emit it.


> * You cannot (easily) enter an RS in vi.


Meh.  !printf '\x1E'.  Also, same answer as above.


> * You cannot see an RS.


But that's harmless.  Especially if you know to expect to find it there.


> * An RS causes Chrome to treat a file as binary data, instead of text.


That could get fixed (e.g., if Chrome learns about this new MIME type).


> * Cut-n-paste a JSON value with an invisible RS prefix and the result is
> NOT JSON, ie it will fail with a JSON parser as RS is not allowed in JSON.


But you can be careful to not cut the RS.  Or we could make it RS SP, to
make it easier to find where to cut.


> * No one uses RS.


That's not much of an argument :)


> * RS is now labelled INFORMATION SEPARATOR TWO, not RECORD SEPARATOR.


Ditto.  And noted.


> * We aren't using INFORMATION SEPARATOR ONE, THREE or FOUR.


Again.


> * A newline as a JSON value terminator is sufficient to parse a JSON
> sequence unambiguously.


Except in the presence of incompletely-written entries.


> * RS doesn't work well with APIs that read text by the line.


Do you have any examples of such APIs?  Define "well".


> * Detecting a newline that separates JSON values is more complex than
> detecting an RS character, but it is not that complex (eg handful of lines
> of code).


Maybe, and that is my preferred solution.  I'm with you there.


> * An RS prefix detects only slightly more cases of accidentally truncated
> writes (in the middle of a top-level number, in a top-level string in the
> middle of an escape sequence) -- not enough to be compelling.


There are other cases if we have as a goal to recover starting at the very
next full text following the truncated one.  Preceding and following texts
with LF also helps recovery in the cases you mention.  But it's not enough.
 The current I-D covers some alternatives that do remove all ambiguities at
little cost to either the parser or the encoder, including: removing
internal newlines from texts (cheap operation; no need to parse and
re-encode) and/or preceding texts with "null" LF.


> * The awkwardness of RS will mean many implementations will be lenient,
> but leniency becomes "expected" which leads to interop problems.


Parsers shouldn't require it.  Encoders should emit it if there's a chance
of truncated writes.  How can interop problems arise from this formula?
 What am I missing?


> "A JSON sequence is the concatenation of zero or more JSON values, where
> each JSON value is terminated with a newline."
>
> Simple to understand. Simple to write. Simple enough to parse. Simple
> enough to resync from the middle of a sequence. Almost identical recovery
> from accidental corruption is possible in almost all the same instances
> regardless of whether an RS prefix or newline suffix is used.
>

Yes, and I very much like that, right up until one wants to cater to
logfiles and the truncated write problem.  See alternatives above.  We must
choose one.  Two camps are squared off and I'm in the middle.  One will win
or all may lose.

My preference is to say that logfile writers must remove internal newlines
from the texts they write.  That is by far the simplest fix for write
truncation.   Not all sequences will be logfile-like.  No RS if we go with
that.  I haven't seen any strong arguments against that, in fact  You have
stronger arguments against RS than I have seen against internal newline
removal by logfile writers.  Let's sleep on it and revisit tomorrow,

Nico
--