Re: [Json] Using a non-whitespace separator (Re: Working Group Last Call on draft-ietf-json-text-sequence)

[Nico Williams <[e-mail]>]

On Wed, Jun 4, 2014 at 1:01 PM, Phillip Hallam-Baker
<[e-mail]> wrote:
> On Wed, Jun 4, 2014 at 1:54 PM, Tim Bray <[e-mail]> wrote:
>> Hah, I hadn’t realized that RS (U+001E, INFORMATION SEPARATOR TWO) was
>> excluded.   OK, so the abnf for JSON-sequence becomes one of these two:

Someone (Carsten?) proposed it earlier.  RS is, indeed, perfect for this.

>>
>> JSON-sequence = JSON-text *( %1e JSON-text )
>> JSON-sequence = *( ws %1e JSON-text )
>>
>> Depending on whether you see the RS as an initiator or a separator.  I think
>> I very slightly prefer the second.

It has to be an initiator for append-write logfiles, as it then marks
the end of a possibly incompletely-written text.  Otherwise you might
lose the first text following an incompletely-written text.

RS could also follow a text, but I prefer LF for this because it's
friendly to line-oriented text tools.  It's harmless to have a
trailing LF (since the JSON-text ABNF allows extra trailing whitespace
anyways).

> +1
>
> I prefer the 'strict writer, lose reader' approach here. And that is
> not my usual stance.

Is that in reference to my (2) or Tim's point?  Since you top-posted,
and since Tim didn't propose a loose reader, I can't tell :)

> The reason that I think readers need to be tolerant is that they
> should be able to read log files after they have been 'damaged' by
> tools that strip out the RS characters.

Yes.  If you see something like:

<text> <text> <text>

it should parse, even though it should have been:

RS<text>RS<text>RS<text>

because, why not parse it?

Thus my (2).

To repeat myself, jq doesn't insist on any separator at parse time,
except where the separator is needed to disambiguate.  E.g., if you
have to texts consisting of numbers, or booleans, or null, then the
parser can't parse them without a separator of some sort.

> For example, lets say that I have some program that records every
> transaction to a logfile and it is discovered that one of the
> transactions was wrong and is corrupting the database. The simplest
> solution is usually to take the log file, find the broken transaction,
> edit it out and rebuild the data base.

Or leave it in and just skip past it when you parse.

> Given the quality of editing tools available on many machines, I don't
> trust them to preserve non printing ASCII characters. Heck, the editor
> on ubuntu can't even start in a root account without writing garbage
> to the terminal.
>
> So readers should be tolerant.

Right, which is one reason that I want the ABNF for writers to be:

    sequence = RS JSON-text LF

but parsers should be more liberal.

I'll grant that if there's an RS present then one can use any kind of
JSON parser to parse the sequence whereas otherwise only incremental
and streaming JSON parsers can be used.  This is the one reason to
require that RS always be written.

Nico
--