Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-06.txt

[Tim Bray <[e-mail]>]

On Sep 30, 2023 at 6:53:28 PM, "Manger, James" <
[e-mail]> wrote:

> Why is U+D800 so much more crucial than U+20FFFF or an ill-formed UTF-8
> byte sequence such as 0xC0 0x80?
>
>
>
> There are lots of byte sequences that are ill-formed UTF-8. Some
> correspond to non-minimal-length encodings (eg C0 80 ~ U+0000); some to
>

Because surrogates are more of a problem in the wild.  My belief is that
this is mostly a consequence of JSON’s \u escapes and Java’s “char”
primitive (OMG I just checked and the official Oracle’s official Java
tutorial still in 2023 says “The char data type is a single 16-bit Unicode
character.”) Plus the JavaScript/browser behavior Rob Sayre describes.

> the explanation of the problem works better if you start from code points
>
>
>
> I disagree. Problems include:
>

I think that both I and the authors of the Unicode standard disagree with
you. The narrative starts with code points and builds up from there.
Unfortunately, this argument can’t be settled by providing technical
evidence, it’s a matter of pedagogical technique.


>    1. Arbitrary code unit sequences can be ill-formed
>
>  > motivate why scalars exist
>
>
>
> Explaining the 1,081,344 size and the U+D800-U+DFFF gap would be
> interesting.
>

Yes! That history is mystery to me.  Also the nonchars in the
Arabic-extended region.

> I don’t think the discussion about scalars and UTF-8/16/32 is
> appropriate. UTF-16 and UTF-32 shouldn’t be used in protocols, the Internet
> has converged on UTF-8
>
>
>
> Argh!
>
> Sure, only use UTF-8 in protocols. But there is also memory & APIs where
> 16-bit code units (hence UTF-16) are still important.
>

You are correct. But this document is designed to help protocol designers
in the IETF, who really have no business straying outside the boundaries of
UTF-8.

If we are only interested in UTF-8 why are surrogates ever mentioned?
>

Because they are forbidden in theory, occur in practice, and break software.


>