Re: [Json] Unpaired surrogates in JSON strings

[Douglas Crockford <[e-mail]>]

On 6/5/2013 11:01 AM, Paul Hoffman wrote:
> <definitely no hat>
>
> On Jun 5, 2013, at 10:46 AM, Douglas Crockford <[e-mail]> wrote:
>
>> On 6/5/2013 9:22 AM, John Cowan wrote:
>>> RFC 4627 Section 1 says:
>>>
>>>      A string is a sequence of zero or more Unicode characters.
>>>
>>> However, the grammar of strings permits things like "foo\uDC00bar",
>>> which contains an escape sequence that does not correspond to any
>>> Unicode character.  This provides backward compatibility with JavaScript,
>>> where a string is not a sequence of characters but a sequence of UTF-16
>>> code units.
>>>
>>> If Section 1 is normative, then there is a contradiction with Section 4,
>>> which says:
>>>
>>>      A JSON parser MUST accept all texts that conform to the JSON
>>>      grammar.
>>>
>>> In my view, JSONbis processors should be REQUIRED to produce only strings
>>> that conform to Section 1.
>>>
>> Such a requirement will be breaking. Breaking changes are out of scope.
> How is that "breaking"? Section 1 has a definition of strings, and Section 4 says that the parser must accept all texts that conform to the grammar. Surrogate code points are not characters, according to the Unicode spec.
>
>> I like the suggestion that section 1 should be talking about code points instead of characters.
> That seems like a significant change that would cause parsers that currently follow Section 1 to fail.
>
It is not a change, it is a clarification. JavaScript, Java, and many 
other languages have strings of code points, owing to their being set 
before Unicode grew surrogate pairs. So strings in those languages are 
composed of code points. JSON is tolerant of that reality.