Re: [Json] Unpaired surrogates in JSON strings

["Martin J. Dürst" <[e-mail]>]

On 2013/06/06 23:57, Tim Bray wrote:
> F0, 90, 8D, 86
> On Thu, Jun 6, 2013 at 4:15 AM, Douglas Crockford<[e-mail]>wrote:
>
>> What  then is the standard name for a 16-bit element of text? When
>> JavaScript was created, that word was character. What is the word now?
>>
>
> The only somewhat-standardized term would be “UTF-16 codepoint”.  But
> that’s not really a “unit of text” any more than the 2nd byte of a
> character encoded in 3 bytes with UTF-8 is.
>
> I’m fairly shocked.  I have always believed that JSON encodes what its
> introduction (and section 2.5 "Strings") say it encodes, Unicode
> characters.
>
> If it is a requirement to accommodate the class of bug where languages that
> use UTF-16 (Java, JavaScript, C#) can emit unpaired UTF-16 surrogates, the
> spec needs to be clear that the INTENT is actually to support Unicode
> characters, and that unpaired surrogates are always evidence of a bug, and
> there can be no expectation that any software receiving such buggy data
> will be able to do anything useful with it, or even avoid crashing in a
> hard-to-debug way down in the bowels of a library routine.  -T

I fully agree with what Tim says above: We know (and to a certain extent 
have to accept) that there are implementations that, surely way more by 
accident than by any kind of intent, send unpaired surrogates. But we 
should try to do whatever we can in the spec to make it perfectly clear 
that there are no good reasons whatsoever to actually do that.

Although we may not end up with exactly parallel or equivalent language, 
I think the situation is fairly similar to the one regarding duplicate 
keys: The current spec isn't totally clear, and in practice it happens, 
and for some implementations, it may be an unreasonable burden to 
require to check everything on sending, but it's not something that one 
would or should do on purpose.

A second point:

Just to get the correct definitions from the Unicode side, here are the 
easiest references, everything on a page:

http://www.unicode.org/glossary/

In a few of my mails today, I have also used "code point", but I didn't 
intend to include surrogates.

There's a better term, namely Unicode Scalar Value 
(http://www.unicode.org/glossary/#unicode_scalar_value):

Any Unicode code point except high-surrogate and low-surrogate code 
points. In other words, the ranges of integers 0 to D7FF16 and E00016 to 
10FFFF16 inclusive. (See definition D76 in Section 3.9, Unicode Encoding 
Forms.)

Regards,    Martin.