[Json] FW: Call for Consensus: Proposed Text for "8.1 Character Encoding"

["Manger, James" <[e-mail]>]

> Recipients that wish to support Unicode encodings other than UTF-8
> can do this using a detection mechanism

Not quite. The detection only distinguishes UTF-8/16/32, not "encodings other than UTF-8".

> that is based on the fact
> that the first character will always have a Unicode code point
> greater than 0

Not quite. It is based on the fact that the (unescaped) character U+0000 cannot appear *anywhere* in valid JSON, and all valid first characters have code points less than 128.

> by inspecting the first octets for nulls.

Might as well be explicitly that inspecting up to the first 4 octets is sufficient. "null" is a special term in JSON so re-using it for a 0x00 octet isn't ideal.

> Text encoded in character encodings other than UTF-8,
> UTF-16, or UTF-32 cannot be used with the media type "application/json".

The sections starts with "MUST be encoded in UTF-8, UTF-16, or UTF-32" so this sentence adds nothing. It just slightly muddies the water. Drop it.



I'll go for option 3) MUST encode as UTF-8 where the media type is 'application/json'.
Suggested text:

"""
  8.1. Character Encoding

  When exchanged as bytes, JSON text MUST be encoded in UTF-8.

  Earlier editions also allowed UTF-16 and UTF-32 encodings. However, those
  choices do not produce interoperable JSON as they are not supported by
  many implementations. UTF-16 and UTF-32 encodings of JSON can be distinguished
  from a UTF-8 encoding by the presence of an octet with value 0 within the first 2 octets,
  since valid first characters all have code points less than 128 and unescaped U+0000 characters
  are not allowed anywhere.

  <paragraph on the byte order mark>
"""

--
James Manger


-----Original Message-----
From: json [mailto:[e-mail]] On Behalf Of Matthew A. Miller
Sent: Friday, 24 March 2017 8:38 AM
Cc: [e-mail]
Subject: Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

Our sponsoring AD and I have been discussing this option.  Alexey is
willing to go through another IETF last call if we can get consensus.

To start, to what degree is UTF-8 encouraged?

0) don't encourage UTF-8 more than already is
1) SHOULD encode as UTF-8 for all usages
2) SHOULD encode as UTF-8 where the media type is 'application/json'
3) MUST encode as UTF-8 where the media type is 'application/json'
4) other -- please specify


-----Original Message-----
From: json [mailto:[e-mail]] On Behalf Of Matthew A. Miller
Sent: Friday, 24 March 2017 1:41 AM
To: [e-mail]
Subject: Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

Hello JSONbis,

It looks like we have consensus for the following text for all of
Section 8.1:

"""
JSON text MUST be encoded in UTF-8, UTF-16, or UTF-32 Section 3 of
[UNICODE].  The default encoding is UTF-8, and JSON texts that are
encoded in UTF-8 are interoperable in the sense that they will be
read successfully by the maximum number of implementations; there are
many implementations that cannot successfully read texts encoded in
UTF-16 or UTF-32. Text encoded in character encodings other than UTF-8,
UTF-16, or UTF-32 cannot be used with the media type "application/json".

Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a JSON text.  In the interests of interoperability,
implementations that parse JSON texts MAY ignore the presence of a
byte order mark rather than treating it as an error.

Recipients that wish to support Unicode encodings other than UTF-8
can do this using a detection mechanism that is based on the fact
that the first character will always have a Unicode code point
greater than 0 and less than 128, thus the UTF-16/32 variants can
be detected by inspecting the first octets for nulls.
"""

Please speak now if you have any objections.

Thank you all,

--
- m&m

Matthew A. Miller
JSONbis Chair