Re: [Json] A possible summary of the discussion so far on code points and characters

[Carsten Bormann <[e-mail]>]

On Jun 9, 2013, at 03:24, R S <[e-mail]> wrote:

> We should document what currently works.

A survey of which implementations react to what input in which way would be nice, indeed, but is not the purpose of this spec update.

Changing the JSON spec retroactively to put in a requirement for handling strings in UTF-16 code units so that unpaired surrogates might work more uniformly is something different.

(BTW, your examples show that two JSON implementations handle Unicode non-characters nicely, which is great and probably something to be recommended, but doesn't have anything to do with switching to UTF-16 code units.  Now let's put in a couple of (paired!) surrogates to show how well the code units work:

>>> print(json.loads('{"a": "\ud800\udd51" }')["a"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 0: surrogates not allowed
>>> print(json.loads('{"a": "\\ud800\\udd51" }')["a"])
𐅑

... which demonstrates nicely what I have been saying: Don't put unescaped surrogates into your JSON texts, because there is no equivalence at the UTF-16 code unit level.)

There is only a single place where the UTF-16 legacy of JavaScript shines through in JSON today:

   To escape an extended character that is not in the Basic Multilingual
   Plane, the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair.  So, for example, a string
   containing only the G clef character (U+1D11E) may be represented as
   "\uD834\uDD1E".

And that is OK because it is just a slightly weird representation of the character.

> > (It is not aligned with JSON's main purpose.)
> 
> I am not sure what the rationale for that statement is.

The first sentence of RFC 4627:

Abstract

   JavaScript Object Notation (JSON) is a lightweight, text-based,
   language-independent data interchange format.

Grüße, Carsten