Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

[Julian Reschke <[e-mail]>]

On 2017-03-13 22:06, Matthew Miller wrote:
> Hello JSONbis,
>
> The security directorate review discussion has raised the issue of
> encoding detection.  The original table from RFC 4627 was removed from
> RFC 7159 due to a lack of consensus.  In this latest round, there have
> been a number of comments have been made supporting (and against) adding
> more guidance than is currently present.
>
> The chair asks for a call on the following from the working group:
>
> 1) Does the working group think adding any text on how to detect the
> encoding worthwhile?

Yes.

> 2a) If such text is worthwhile, is the following proposed text from Nico
> Williams acceptable (to be appended to Section 8.1)?
>
> """
>    Implementors MAY count the number of ASCII NULs in the first four

s/MAY/can/

>    bytes of any JSON text to detect which of UTF-8, UTF-16, or UTF-32
>    the text is encoded in:
>
>     - if the count is zero, then the text is encoded in UTF-8
>     - if the count is one or two, then the text is encoded in UTF-16
>     - if the count is three, then the text is encoded in UTF-32
>
>    This results from a) JSON texts having to start with an ASCII
>    character, b) no unescaped NULs being allowed in JSON strings, and c)
>    any type being allowed at the top-level, thus the first character may
>    be a double-quote and the second may be any permissible, unescaped
>    Unicode codepoint.  An ASCII character requires a NUL-valued byte in
>    UTF-16 encoding, three in UTF-32, and none in UTF-8.

...and add that if the number of octets is less than 4, it's UTF-8.

> """
> 2b) If such text is worthwhile but Nico's proposal is not worthwhile,
> what would be acceptable?
>
> Please respond by March 16.
> ...

I personally would prefer the table that I posted, as directly 
identifies byte ordering as well:

>            00 00 00 xx  UTF-32BE
>            00 xx 00 xx  UTF-16BE
>            xx 00 00 00  UTF-32LE
>            xx 00 xx 00  UTF-16LE
>            xx xx xx xx  UTF-8

Best regards, Julian