Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

["Manger, James" <[e-mail]>]

General

On Oct 2, 2023 at 5:20:59 PM, "Manger, James" <[e-mail]<mailto:[e-mail]>> wrote:
draft-bray-unichars<https://datatracker.ietf.org/doc/html/draft-bray-unichars> §3 “Dealing with problematic code points” suggests “replacing problematic code points with "�" (U+FFFD, REPLACEMENT CHARACTER)” (or signalling an error, but I’ll only talk about the replacement option in this email).

  1.  An ill-formed sequence of code units needs to be replaced. It is far less obvious to me that “problematic” scalars should be replaced. Even for noncharacters Unicode provides a good FAQ<https://www.unicode.org/faq/private_use.html#nonchar9> and corrigendum #9 “Clarification about noncharacters”<https://www.unicode.org/versions/corrigendum9.html> that suggests passing them along (treating them like unassigned scalars) is often the best policy (because the internal/interchange boundary is blurry).
> OK, that’s worth a reference.

  1.  So §4.3 defining unicode-assignable that excludes noncharacters is fine -- when to be lenient on receiving a supposed unicode-assignable value is less obvious.
But §3 looks dodgy.
> Would a note that it might be reasonable to accept nonchars, referencing that corregendum, de-dodgify it in your view?

If it might be reasonable to accept nonchars, it presumably might be reasonable to accept controls or any scalar. To de-dodgify, the text should not conflate ill-formed code units with scalars.

“Virtuous intolerance” [RFC9413<https://www.rfc-editor.org/rfc/rfc9413.html#name-virtuous-intolerance>] with respect to UTF-8/16/32 is clear and widely implemented: signal an error; or replace with U+FFFD (or U+003F). Presumably this is why javascript is changing JSON.stringify to always escape an unpaired surrogate (not just accepting escaped-unpaired-surrogates in JSON.parse).

“Virtuous intolerance” with respect to xml-character or unicode-assignable is less clear to me. Maybe it is left to future specs that refers to these repertoires? Or maybe this doc can pick “rules for consistent handling of aberrant conditions”. That means this doc doesn’t merely name some repertoires but adds handling rules. Sounds feasible; could be controversial. Can we pick “always signal an error”? Or do we need to offer “or replace scalars-not-in-the-repertoire with U+FFFD”. It’s just that I’m not sure any systems do the latter.

In any case, I’d like to see any such “virtuous intolerance” rules for this doc’s repertoires described separately from Unicode’s existing “virtuous intolerance” rules for UTF-8/16/32.

--
James Manger