Re: [Json] Proposed minimal change for strings

[John Cowan <[e-mail]>]

Paul Hoffman scripsit:

> >>  Add "Some strings, notably those that have unescaped surrogate code units
> >>  (value 0xD800 to 0xDFFF), cannot be encoded in UTF-8."
> > 
> > Unescaped and *unpaired*.
> 
> No, any surrogate code point. RFC 3629, the IETF's definition of UTF-8, says:
>    The definition of UTF-8 prohibits encoding character numbers between
>    U+D800 and U+DFFF, which are reserved for use with the UTF-16
>    encoding form (as surrogate pairs) and do not directly represent
>    characters.

You are conflating code points (integers in the range 0-0x10FFFF,
definition D10) with code units (bit-strings of a specified length,
definition D77).  The code *points* corresponding to surrogates
(0xD800-0xDFFF) cannot be encoded with any encoding.  The 16-bit code
*units* corresponding to surrogates are used in pairs to represent
characters from U+10000 to U+10FFFF.  There are, obviously, no such
8-bit code units, as the numbers involved will not fit in 8 bits, and
the equivalent 32-bit code units are not used.

Now you make me wonder if your proposal 1 was supposed to be about code
points rather than code units.

(from another email in this thread)

> That appears to be the case for the current document. If you have a
> preferred [code unit] size, by all means propose it to the list. If
> we can get rough consensus on that, it would help this discussion.

In the JSON context, the only code unit size that makes sense is 16-bit,
because JavaScript (from which JSON comes) deals in 16-bit code units,
sequences of which are called "strings".  But if you mean to talk of code
points rather than units, then of course there is no size to specify,
as integers don't have a size.

> > As noted, that should have been "unpaired unescaped surrogate
> > code units".
> 
> That is only true for UTF-16. The definitions of UTF-8 and UTF-32 say
> that you cannot encode any surrogate code points, paired or not.

This is the same conflation.

-- 
John Cowan <[e-mail]>             http://www.ccil.org/~cowan
Sir, I quite agree with you, but what are we two against so many?
    --George Bernard Shaw,
         to a man booing at the opening of _Arms and the Man_