Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

["Manger, James" <[e-mail]>]

draft-bray-unichars<https://datatracker.ietf.org/doc/html/draft-bray-unichars> §3 “Dealing with problematic code points” suggests “replacing problematic code points with "�" (U+FFFD, REPLACEMENT CHARACTER)” (or signalling an error, but I’ll only talk about the replacement option in this email).


  1.  An ill-formed sequence of code units needs to be replaced. It is far less obvious to me that “problematic” scalars should be replaced. Even for noncharacters Unicode provides a good FAQ<https://www.unicode.org/faq/private_use.html#nonchar9> and corrigendum #9 “Clarification about noncharacters”<https://www.unicode.org/versions/corrigendum9.html> that suggests passing them along (treating them like unassigned scalars) is often the best policy (because the internal/interchange boundary is blurry).
So §4.3 defining unicode-assignable that excludes noncharacters is fine -- when to be lenient on receiving a supposed unicode-assignable value is less obvious.
But §3 looks dodgy.
  2.  U+FFFD is an obvious choice to replace code units or scalars you don’t want. But Unicode does allow choices. Unicode ch3<https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf> C10 only says “with a marker such as U+FFFD”. Unicode TR36<https://unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences> says “where U+FFFD is not available, a common alternative is "?"”. Java, for instance, uses “?” is some common circumstances. Unichars does not admit such an option.
  3.  “Silently ignoring” is the wrong phrase. The security risk is “deleting” ill-formed sequences or unwanted scalars. “Silently ignoring” feels the same as “deleting” when decoding code units to scalars; but feels different when processing input chars to output chars as it covers passing along untouched an unliked scalar.

--
James Manger




General

From: Tim Bray <[e-mail]>
Date: Tuesday, 3 October 2023 at 9:40 am
To: Carsten Bormann <[e-mail]>
Cc: Manger, James <[e-mail]>, [e-mail] <[e-mail]>, ART Area <[e-mail]>, Rob Sayre <[e-mail]>
Subject: Re: [art] New Version Notification for draft-bray-unichars-06.txt
[External Email] This email was sent from outside the organisation – be cautious, particularly with links and attachments.
On Oct 2, 2023 at 9:14:18 AM, Carsten Bormann <[e-mail]<mailto:[e-mail]>> wrote:
 The IETF could pound its collective fist and say "all ill-formed Unicode must be rejected”,

Yes, please.
The fact that this is the only reasonable way forward is the point of RFC 9413.

Now we agree! And further (especially given the threats described in Unicode TR36) you often also want to reject control codes and noncharacters.  I think the IETF should be shouting this!

To promote this, it would be helpful if people actually understood what the problems are, and which code points to reject, and had a reference that explained the issues and provided ABNF for what to accept, at increasing levels of fussiness. Then when the IETF starts shouting, there will be a short clean reference to accompany the shouting.