Re: [Json] Unpaired surrogates in JSON strings

[Paul Hoffman <[e-mail]>]

<no hat>

The text says "character" everywhere. It says "code point" exactly once, when it is talking about a character's code point.

The ABNF seems to allow code points, not characters. This is pretty clearly a mistake.

This could be resolved by a warning similar to what Joe proposed, but it should be stronger. Also, Joe's wording assumes that these code points are always represented as escape sequences, but the ABNF allows them to be in a string as raw, unescaped code points.

On Jun 5, 2013, at 4:35 PM, Joe Hildebrand (jhildebr) <[e-mail]> wrote:

> I don't mind allowing surrogate pairs in \u notation.  But I think we
> should specify what happens when you send just half of the pair.  For
> example, adding this after section 2.5, graph 4:
> 
> Escape sequences between \uD800 and \uDFFF SHOULD be generated only as
> valid UTF16 surrogate pairs (this SHOULD is only to allow backward
> compatibility).  When encountering an invalid surrogate pair (such as
> "foo\uD834bar" or "\uDD1E\uD834"), parsers MAY either throw an error
> (taking the risk of some backward incompatibility with old generators) or
> MAY ignore the sequence.

Alternate proposal:

Code points between U+D800 and U+DFFF SHOULD be generated only as
valid UTF16 surrogate pairs; this SHOULD is only to allow backward
compatibility with applications that ignored the restriction that
strings consist of Unicode characters. A parser that encounters
an invalid surrogate pair (such as "foo\uD834bar" or "\uDD1E\uD834"),
SHOULD throw an error because the string does not consist of characters;
it might ignore the errant code points, but at the risk of allowing
strings that other parsers would find illegal.

--Paul Hoffman