Re: [Json] Proposed minimal change for strings

[Nico Williams <[e-mail]>]

On Tue, Jul 2, 2013 at 6:27 PM, Paul Hoffman <[e-mail]> wrote:
> <chair hats on>
>
> Do either or both of these proposals work for people in the WG?
>
> --Matt Miller and Paul Hoffman
>
> The following are proposed minimal changes to make the JSON spec interoperable with respect to
> which unescaped code units are legal in strings.
>
> Proposal 1 (allow all code units in their unescaped form):

Huh?  Do you mean that any code unit may be allowed if escaped?  Or
did you really mean that any code unit may be sent unescaped?
Certainly the latter would be bad (e.g., double-quotes and newline
must be escaped at the very, very least).

> In section 1 (Introduction):
>   Change the sentence about Unicode characters to:
>     A string is a sequence of zero or more Unicode code units [UNICODE].

These aren't Unicode things though.  They are 16-bit values that
usually can be interpreted as part of UTF-16-encoded Unicode text.

> In section 2.2 (Strings):
>   Leave the production for "unescaped" as-is.
> In section 3 (Encoding):
>   Add "Some strings, notably those that have unescaped surrogate code units
>   (value 0xD800 to 0xDFFF), cannot be encoded in UTF-8."

Unescaped and *unpaired*.

Note that parsers can unescape escaped characters/code points/code
units, and many do.  Therefore banning unescaped unpaired surrogate
halves is not sufficient.  We must say something about what parsers
SHOULD do when they see these, whether escaped or unescaped.

> Proposal 2 (prohibit unescaped surrogates):
>
> In section 1 (Introduction):
>   A string is a sequence of zero or more Unicode scalar values [UNICODE].
> In section 2.2 (Strings):
>   Change the production for "unescaped" to be:
>     unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

I don't think we need to make this change: parsers can be expected to
do something with unpaired surrogate halves, like either throw them
away or escape them (if it makes sense in whatever context the parser
is executing in) or even output a binary string instead of a text
string.  Certainly, if we don't make this change, then we should say
something about what parsers SHOULD do in the face of such things.
Anyways, I'd +1 this, but we need to discuss what to do with these
when they are escaped.

Nico
--