Re: [Json] Proposal for strings/Unicode text

[Nico Williams <[e-mail]>]

On Thu, Jun 13, 2013 at 1:00 PM, Tim Bray <[e-mail]> wrote:
> On Thu, Jun 13, 2013 at 5:16 AM, John Cowan <[e-mail]> wrote:
>> The point is that if JSON is encoded in UTF-8, any surrogate code points
>> MUST be escaped, even though the grammar does not say so.
>
> Why?  UTF-8 is perfectly capable of representing those integers.  Yes, the
> spec says that You Shouldn’t Do That, but it says the same thing about
> unpaired surrogates in UTF-16.  For historical reasons JSON allows the
> encoding of stuff that is strictly nonconforming to Unicode.  This will
> break lots of things, not just UTF-8 decoders (most of which, I bet, will
> never actually notice).  -T

I was thinking the same thing.  We're way into "you shouldn't do that
but actually, look, you can" territory.

I think it's important that some \uXXXX sequences not be unescaped by
intermediaries, or, rather, that certain code points always be escaped
by encoders.  E.g., \u0000, \u2028, \uD800, and so on.  The reason is
that a parser may not be able to handle such code points unescaped.
We should stop at that and otherwise allow just about any code point.

Nico
--