Re: [Json] Proposal for strings/Unicode text

[John Cowan <[e-mail]>]

Bjoern Hoehrmann scripsit:

> >Officially, yes.  But surrogate code points cannot be inserted directly
> >if the representation is UTF-8 (otherwise it becomes CESU-8 instead)
> >or UTF-16 (otherwise it is broken UTF-16) or random non-Unicode encodings.
> >So UTF-32 is the only encoding into which a surrogate code point can be
> >inserted directly -- and nobody uses it.
> 
>     • Because surrogate code points are not included in the set of 
>       Unicode scalar values, UTF-32 code units in the range
>       0000D800_16 .. 0000DFFF_16 are ill-formed.

Well, sure.  Note the fine distinction between "can" (in physical
fact) and "may" (in the RFC 2119 sense).  It's invalid to have unpaired
surrogates in *any* context.  But at least it is safe and possible to do
so in UTF-32.  In UTF-8, there is no representation at all, and in UTF-16
you can't tell the difference between two consecutive unpaired surrogates
of opposite polarities and a surrogate pair.  (Though come to think of
it, escaping doesn't allow two consecutive unpaired surrogates either,
so maybe we can fairly say that either UTF-16 or UTF-32 allow them.)

The point is that if JSON is encoded in UTF-8, any surrogate code points
MUST be escaped, even though the grammar does not say so.

-- 
John Cowan            http://www.ccil.org/~cowan     [e-mail]
Uneasy lies the head that wears the Editor's hat! --Eddie Foirbeis Climo