Re: [Json] Wording on encoding; removing the table

I believe we must have consensus that this is a contentious issue, and there 
is a lot of confusion around it.  Therefore, in the interests of 
interoperability I believe it is inappropriate to decide to be silent on all 
of these issues.  Therefore, I propose text along the following lines:

    JSON text is a sequence of Unicode codepoints.  The transfer encoding
    used to represent the characters on-the-wire is beyond the scope
    of this document.  It is therefore up to the specifications that
    reference this document to specify whether JSON messages will be
    transferred using UTF-8 (recommended), UTF-16 and/or UTF-32
    (discouraged), and whether preceding BOMs must be present,
    must not be present or are optional.

    If multiple encodings are permitted, implementers may choose to
    auto-detect a message's encoding by exploiting the fact that the
    first character of a JSON text must be in the ASCII character
    range and use the following table to deduce the active encoding:

           xx xx -- --  UTF-8
           xx 00 xx --  UTF-16LE
           xx 00 00 xx  UTF-16LE
           xx 00 00 00  UTF-32LE
           00 xx -- --  UTF-16BE
           00 00 -- --  UTF-32BE

I don't think that's a lot of text with which to describe the issues here, 
and I'm sure Tim (or someone else) can make it even snappier and more 
accurate.

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
----- Original Message ----- 
From: "Paul Hoffman" <paul.hoffman@vpnc.org>
To: "JSON WG" <json@ietf.org>
Sent: Friday, November 22, 2013 10:36 PM
Subject: [Json] Wording on encoding; removing the table

> <hat on>
>
> Please note that the chairs tried to find some consensus in the BOM
> discussion and found none. Given that, and given that the current table is
> now wrong, our proposal is to remove it, not try to doctor it.
>
> Current Section 8.1:
>
>   JSON text SHALL be encoded in Unicode.  The default encoding is
>   UTF-8.
>
>   Since the first two characters of a JSON text will always be ASCII
>   characters [RFC0020], it is possible to determine whether an octet
>   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
>   at the pattern of nulls in the first four octets.
>
>   00 00 00 xx  UTF-32BE
>   00 xx 00 xx  UTF-16BE
>   xx 00 00 00  UTF-32LE
>   xx 00 xx 00  UTF-16LE
>   xx xx xx xx  UTF-8
>
> Proposed replacement:
>
>   The default encoding for JSON transmitted over the Internet is UTF-8.
>   Transmitting JSON using other encodings may not be interoperable
>   unless the receiving system definitively knows the encoding.
>
> Does anyone have a technical objection to the proposed replacement? If so,
> please state the error and (hopefully) a correction.
>
> --Matt Miller and Paul Hoffman
>
> _______________________________________________
> json mailing list
> json@ietf.org
> https://www.ietf.org/mailman/listinfo/json