Re: [Json] Proposal: Code points and surrogates

[Norbert Lindenberg <[e-mail]>]

On Jun 22, 2013, at 7:58 , Stephen Dolan wrote:

> On Tue, Jun 18, 2013 at 8:43 AM, Norbert Lindenberg
> <[e-mail]> wrote:
>> Before:
>>    A string is a sequence of zero or more Unicode characters [UNICODE].
>> 
>> After:
>>    A string is a sequence of zero or more Unicode code points [UNICODE].
> 
> Unicode's three character encoding forms (UTF-8, UTF-16, and UTF-32)
> define means of encoding sequences of unicode scalar values. A unicode
> scalar value is a code point which is not a surrogate. Noncharacter
> codepoints like U+FFFF are unicode scalar values.
> 
> Since JSON is invariably represented in some unicode encoding, it
> seems odd that JSON would include things which are not representable
> in any standard unicode encoding.
> 
> Should we instead say "a string is a sequence of zero or more Unicode
> scalar values"? Even if we use "code point", it still seems odd to
> claim "this specification allows the inclusion of surrogate code
> points (U+D800 through U+DFFF) in JSON text, both directly and through
> escape sequences" - it's not clear that the inclusion of surrogate
> code points directly is actually possible.

They can be included in Unicode strings, as defined in the Unicode Standard, section 2.7 [1]. The strings used in JavaScript, Java, and some other languages are Unicode strings, with no requirement of well-formedness.

The specification of the JSON object in the ECMAScript Language Specification [2] requires that unpaired surrogates in strings are passed through by both parse() and stringify(), and that parse() unescapes escape sequences for all surrogate values, paired or not. Testing I've done indicates that the implementations in the major browsers and in Node.js conform to this requirement.

[1] http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf
[2] http://ecma-international.org/ecma-262/5.1/#sec-15.12

Norbert