Re: [Json] [secdir] secdir review of draft-ietf-jsonbis-rfc7159bis-03

[Nico Williams <[e-mail]>]

On Mon, Mar 13, 2017 at 04:51:58PM +0900, Martin J. Dürst wrote:
> My personal opinion is that we could try to fix this by changing the
> following:
> 
> >>>>
>    JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32 [UNICODE]
>    (Section 3).  The default encoding is UTF-8, and JSON texts that are
>    encoded in UTF-8 are interoperable in the sense that they will be
>    read successfully by the maximum number of implementations; there are
>    many implementations that cannot successfully read texts in other
>    encodings (such as UTF-16 and UTF-32).
> >>>>
> 
> to something like the following:
> 
> >>>>
>    JSON text SHOULD be encoded in UTF-8 [UNICODE]
>    (Section 3).  JSON texts that are
>    encoded in UTF-8 are interoperable in the sense that they will be
>    read successfully by the maximum number of implementations.
> 
>    There are
>    many implementations that cannot successfully read texts in other
>    encodings (such as UTF-16 and UTF-32). JSON text MAY be encoded in
>    UTF-16 or UTF-32 [UNICODE] (Section 3) if the sender is sure that
>    the intended recipients can read them.
> >>>>
> 
> That should then go together with a MIME registration that only lists UTF-8.

+1.

I would restore the text from an older draft about how counting the
number of NULs in the first four bytes can be used to determine which of
UTF-8, 16, or 32 the text is encoded in, modified to avoid all talk of
BOMs (which had no consensus):

   Implementors MAY count the number of ASCII NULs in the first four
   bytes of any JSON text to detect which of UTF-8, UTF-16, or UTF-32
   the text is encoded in:

    - if the count is zero, then the text is encoded in UTF-8
    - if the count is one or two, then the text is encoded in UTF-16
    - if the count is three, then the text is encoded in UTF-32

   This results from a) JSON texts having to start with an ASCII
   character, b) no unescaped NULs being allowed in JSON strings, and c)
   any type being allowed at the top-level, thus the first character may
   be a double-quote and the second may be any permissible, unescaped
   Unicode codepoint.  An ASCII character requires a NUL-valued byte in
   UTF-16 encoding, three in UTF-32, and none in UTF-8.

Note the "MAY".  The idea is that we should be sending (and storing)
JSON texts encoded in UTF-8 per the text I +1'ed above, so we shouldn't
bother recommending or requiring the use of this UTF detection scheme.
But there's no harm in describing it.

See https://www.ietf.org/mail-archive/web/json/current/msg02053.html

The encoding detection algorithm can be optimized a bit since if neither
the first nor the second bytes are NUL-valued then the encoding must be
UTF-8.

The algorithm can also be combined with an understanding of byte
ordering to detect encoding by looking only at the first two bytes in
some cases, but whatever.

ASIDE:

  I don't see any reason not to also include text discussing byte-order
  detection in the UTF-16 and UTF-32 cases, but I don't care very much
  about it, and given the prior lack of consensus for it, there's no
  need to relitigate that point now.  Also, it's pretty obvious how to
  detect byte-order anyways, at least given the text on how to detect
  encoding: just looking at the first two bytes will tell you what the
  byte order is!

  If the first byte is zero-valued, then the byte order is big-endian,
  if the first is not zero-valued but the second is, then the byte order
  is little-endian, and if both, the first and second bytes are
  zero-valued, then the byte order is big-endian; if neither the first
  nor the second bytes are zero-valued then the encoding is UTF-8 and
  byte-order is irrelevant.

Cheers,

Nico
--