Re: [Json] Encoding Schemes

[Carsten Bormann <[e-mail]>]

On Jun 18, 2013, at 21:34, Paul Hoffman <[e-mail]> wrote:

>> JSON the media type (application/json) is specifically limited to UTF-8 (and theoretically the two or possibly four other character encoding schemes listed in RFC 4627; the RFC isn't quite consistent here).
> 
> Can you point to the text in the draft that supports that statement? I see the opposite:
>   Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32
> 
>     JSON may be represented using UTF-8, UTF-16, or UTF-32.  When JSON
>     is written in UTF-8, JSON is 8bit compatible.  When JSON is
>     written in UTF-16 or UTF-32, the binary content-transfer-encoding
>     must be used.

You listed two of the three places, the third is in section 3, which doesn't really list UTF-16 but UTF-16 "BE or LE" (same for UTF-32).  Note that these are three different character encoding schemes (CESs), so it is not clear which ones are actually meant.


   Since the first two characters of a JSON text will always be ASCII
   characters [RFC0020], it is possible to determine whether an octet
   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
   at the pattern of nulls in the first four octets.

           00 00 00 xx  UTF-32BE
           00 xx 00 xx  UTF-16BE
           xx 00 00 00  UTF-32LE
           xx 00 xx 00  UTF-16LE
           xx xx xx xx  UTF-8


[Obligatory Unicode bashing: Giving one of the three encoding schemes (CESs) for the encoding form (CEF) UTF-16 the same name, i.e., "UTF-16", must have been decided by a very cruel person.]

(Strictly speaking, the other mentions of UTF-16/-32 might be the encoding form, not the scheme, but the RFC simply isn't completely specified here.  I think the only reasonable way to read this has the same result as what Joe is proposing, but some text interpretation is required.  Clearly, the text in section 3 does not work with the BOM-based CESs.  But you might also read the text in 6 as asking for UTF-16 CES and the text in 3 then excluding BOMs so the UTF-16 CES is implicitly big-endian.  So much effort for something so theoretical.)

Grüße, Carsten