Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-06.txt

[Rob Sayre <[e-mail]>]

On Sat, Sep 30, 2023 at 6:53 PM Manger, James <James.H.Manger=
[e-mail]> wrote:

>
>
> > the explanation of the problem works better if you start from code
> points
>
>
>
> I disagree. Problems include:
>
>    1. Arbitrary code unit sequences can be ill-formed
>    2. BOM
>    3. Noncharacters
>    4. Legacy controls
>    5. Tag chars and other deprecated format chars
>    6. NFC vs NFKC; canonical chars
>    7. Private use chars
>    8. BIDI
>    9. …
>    10. Escaping
>
>
>
> #1 is about encodings. #2 is about encodings and is a char. #3-7 are about
> char subsets. #10 is about a higher-level.
>
> Code points don’t help explain any of these.
>

Well, the grammar describes code points, and the document needs to explain
the unassigned ones as well as the surrogate code points.

Some of the topics you raise here are things that people writing Unicode
libraries need to know about, but are fairly irrelevant to Unicode users.
Most people will be calling something like std::str::from_utf8 [0] if
they're not using some higher-level library for JSON or CBOR etc. At the
limit, people might ask themselves "what's the ICU function for this?", but
they're beyond the scope of this document at that point.

Normalization actually does come up in application code sometimes. For
example, tweets need to be NFC to count characters correctly [1]. But the
person writing that code is not going to learn much from this document (I
do find the subsets useful and concise, though).

Unpaired surrogate code points and their escaped form need to be covered,
since the standard behavior of billions of web browsers is to smuggle them
in well-formed UTF-8 via escape sequences. [2]

thanks,
Rob

[0] https://doc.rust-lang.org/std/str/fn.from_utf8.html
[1]
https://github.com/sayrer/twitter-text/blob/cxx/rust/twitter-text/src/extractor.rs#L323C63-L323C63
[2] https://github.com/tc39/proposal-well-formed-stringify