Re: [Json] Using a non-whitespace separator (Re: Working Group Last Call on draft-ietf-json-text-sequence)

[Nico Williams <[e-mail]>]

On Sun, Jun 1, 2014 at 6:17 PM, Paul Hoffman <[e-mail]> wrote:
>>> Oh, right, the separator must be a character that must be escaped in
>>> strings.  That greatly limits the range of characters we can choose
>>> from.
>>
>> And it has to be a one-byte character (therefore an ASCII character,
>> and the texts must be encoded in UTF-8).
>
> Why is that?

The problem we're talking about is logfiles where applications
"append" to the logfile.  Incomplete writes can result in some
circumstances.  The question then is: how to recover?  In particular:
how to read past an incomplete entry to the next complete entry?

The I-D currently describes a recovery heuristic, but some reviewers
have stated a preference for a stronger, more easily understood
recovery method.  One obvious approach is to separate texts with a
byte or byte sequence that could not normally happen in a JSON text.
A byte is simpler and easier to handle and understand than a byte
sequence: because the latter can be written incompletely for the same
reasons that a JSON text can.

We've considered all of these approaches (not in this order):

0) newline separator
1) #0 + JSON text boundary detection ABNF
2) #1 + removal of newlines from JSON texts (which does not require
re-encoding, FYI)
3) #2 + write something like a text consisting of a single null value
before every text
4) use some other separator (that isn't a JSON whitespace character)
5) #4 with ASCII RS as the separator

Any separator for #4 has to be something that cannot happen in a JSON
text normally, and it has to be something amenable to recovery from
partial writes.  Even partial writes always at least write complete
_bytes_ (if they write any).  Therefore a one-byte separator is always
unambiguous.

In order for a separator to be utterly unambiguous in the face of
partial writes it has to involve a byte that can never occur in a JSON
text.  I suspect there's no such byte that works for UTF-8/16/32.  If
we limit JSON text sequences to UTF-8 then any ASCII character that
must be escaped in strings and is not valid in the encoding of any
values will do.  That's a very small set of characters.  RS (#5) was
objected to.  I don't see what other character can be used that won't
result similar objections.

The more I think about it the more I prefer the options in the I-D,
#0, #1, #2, and #3.  I don't see better alternatives, and I don't
think this is fatal.

Nico
--