[Json] Encoding Schemes

["Joe Hildebrand (jhildebr)" <[e-mail]>]

Background: the word "encoding" is somewhat imprecise.  From the Unicode
Glossary (http://www.unicode.org/glossary/):

"""
Character Encoding Scheme. A character encoding form plus byte
serialization. There are seven character encoding schemes in Unicode:
UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE.


Unicode Encoding Form. A character encoding form that assigns each Unicode
scalar value to a unique code unit sequence. The Unicode Standard defines
three Unicode encoding forms: UTF-8, UTF-16, and UTF-32.

Unicode Encoding Scheme. A specified byte serialization for a Unicode
encoding form, including the specification of the handling of a byte order
mark (BOM), if allowed.
"""

In UTF-16 and UTF-32 (note: *not* BE or LE), you SHOULD include a Byte
Order Mark, but if it's not there, you assume BE.  From Unicode 6.2,
section 3.10, D98:

"""
The BOM is not considered part of the content of the text.


The UTF-16 encoding scheme may or may not begin with a BOM. However,
when there is no BOM, and in the absence of a higher-level protocol, the
byte
order of the UTF-16 encoding scheme is big-endian.
"""

and from D101:

"""
The BOM is not considered part of the content of the text.

The UTF-32 encoding scheme may or may not begin with a BOM. However,
when there is no BOM, and in the absence of a higher-level protocol, the
byte
order of the UTF-32 encoding scheme is big-endian.
"""


One could argue that the current table in section 3 of 4627 constitutes
such a higher-level protocol, but since we haven't specified either UTF-16
or UTF-32 in the table, nor have we specified what to do with BOMs, I
don't think that 4627 currently acts as that higher-level protocol.

As such, I believe that the phrase from section 3 of 4627 that says:


"JSON text SHALL be encoded in Unicode.  The default encoding is UTF-8."

needs a little help.  We probably want to include only the 5 encoding
schemes currently mentioned in the draft, not all 7.  Some suggest text:

"When serialized to an octet stream, JSON text SHALL be encoded in one of
the following Unicode encoding schemes: UTF-8,  UTF-16BE, UTF-16LE,
UTF-32BE, and UTF-32LE.  The default and RECOMMENDED encoding is UTF-8.
Byte Order Marks MUST NOT be used to signal byte order in any encoding."

and then on with the changed version of the table that we've discussed.
This text intentionally allows U+FEFF in strings (where they've been
allowed to date as zero width no-break space), but NOT as the first
codepoint in the stream, where, since U+FEFF isn't in the ws production,
they can't be used.

As well, if Unicode adds another encoding scheme one day, I don't want
4627bis to automatically require support for it unintentionally.

-- 
Joe Hildebrand