Re: [I18ndir] draft-bray-unichars-01

[Nico Williams <[e-mail]>]

On Wed, Aug 30, 2023 at 09:48:21AM -0700, Tim Bray wrote:
> On Aug 29, 2023 at 12:10:42 PM, Asmus Freytag <[e-mail]> wrote:
> > This subset has the advantage of excluding surrogates, which can never add
> > any value and have the potential to cause problems.
> >
> > should be reworded a bit to:
> >
> > "This subset has the advantage of excluding surrogates, which are not
> > assigned to any characters, and thus can never add any value.
> > They have the potential to cause problems, for example it is not possible
> > to represent them individually in UTF-8."
> 
> The Unicode Standard says that surrogates can only be used in UTF-16 and
> RFC3629 explicitly says they are prohibited. Having said that, it’s easy to
> generate UTF-8 that includes them (start by truncating a Java “char” array
> incautiously, or just have a bad day while programming in C) and most
> (all?) language implementations per Postel’s law will happily give you that
> surrogate code point. That’s the problem; although it’s forbidden it can be
> done and it does happen.  So I think that instead of “not possible” we
> should say something like “… potential to cause problems; while it is
> possible to include them in UTF-8 strings, this is forbidden by the
> specification of UTF-8”.

You can always use a JSON escape sequence to represent an unpaired
surrogate codepoint.  And who knows, maybe on parsing it will be
unescaped but not substituted.

Ideally no software should produce even escaped unpaired surrogates, and
ideally all parsers should replace or escape non-UTF-8, and/or maybe
replace escapes that are not valid Unicode.

(It's not even clear that JSON parsers should unescape escapes.  RFC
8259 is silent on this matter.  The text in section 8.2 seems to hint
that a parser should either unescape to the max or escape to the max,
but then section 8.3 more strongly implies no requirement.)

Nico
--