Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

[Tim Bray <[e-mail]>]

 [inline]

On Oct 2, 2023 at 5:20:59 PM, "Manger, James" <
[e-mail]> wrote:

> draft-bray-unichars
> <https://datatracker.ietf.org/doc/html/draft-bray-unichars> §3 “Dealing
> with problematic code points” suggests “replacing problematic code points
> with "�" (U+FFFD, REPLACEMENT CHARACTER)” (or signalling an error, but I’ll
> only talk about the replacement option in this email).
>
>
>
>    1. An ill-formed sequence of code units needs to be replaced. It is
>    far less obvious to me that “problematic” scalars should be replaced. Even
>    for noncharacters Unicode provides a good FAQ
>    <https://www.unicode.org/faq/private_use.html#nonchar9> and corrigendum
>    #9 “Clarification about noncharacters”
>    <https://www.unicode.org/versions/corrigendum9.html> that suggests
>    passing them along (treating them like unassigned scalars) is often the
>    best policy (because the internal/interchange boundary is blurry).
>
> OK, that’s worth a reference.


>    1. So §4.3 defining unicode-assignable that excludes noncharacters is
>    fine -- when to be lenient on receiving a supposed unicode-assignable value
>    is less obvious.
>    But §3 looks dodgy.
>
> Would a note that it might be reasonable to accept nonchars, referencing
that corregendum, de-dodgify it in your view?

>
>    1.
>    2. U+FFFD is an obvious choice to replace code units or scalars you
>    don’t want. But Unicode does allow choices. Unicode ch3
>    <https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf> C10 only
>    says “with a marker such as U+FFFD”. Unicode TR36
>    <https://unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>
>    says “where U+FFFD is not available, a common alternative is "?"”. Java,
>    for instance, uses “?” is some common circumstances. Unichars does not
>    admit such an option.
>
> Also worth a reference. If you’re writing Java code you should probably do
what Java does, no?


>    1.
>    2. “Silently ignoring” is the wrong phrase. The security risk is
>    “deleting” ill-formed sequences or unwanted scalars. “Silently ignoring”
>    feels the same as “deleting” when decoding code units to scalars; but feels
>    different when processing input chars to output chars as it covers passing
>    along untouched an unliked scalar.
>
> Sounds about right.


>    1.
>
>
>
> --
>
> James Manger
>
>
>
>
>
>
>
> * General From: *Tim Bray <[e-mail]>
> *Date: *Tuesday, 3 October 2023 at 9:40 am
> *To: *Carsten Bormann <[e-mail]>
> *Cc: *Manger, James <[e-mail]>, [e-mail] <
> [e-mail]>, ART Area <[e-mail]>, Rob Sayre <[e-mail]>
> *Subject: *Re: [art] New Version Notification for
> draft-bray-unichars-06.txt
>
> [External Email] This email was sent from outside the organisation – be
> cautious, particularly with links and attachments.
>
> On Oct 2, 2023 at 9:14:18 AM, Carsten Bormann <[e-mail]> wrote:
>
>  The IETF could pound its collective fist and say "all ill-formed Unicode
> must be rejected”,
>
>
> Yes, please.
> The fact that this is the only reasonable way forward is the point of RFC
> 9413.
>
>
>
> Now we agree! And further (especially given the threats described in
> Unicode TR36) you often also want to reject control codes and
> noncharacters.  I think the IETF should be shouting this!
>
>
>
> To promote this, it would be helpful if people actually understood what
> the problems are, and which code points to reject, and had a reference that
> explained the issues and provided ABNF for what to accept, at increasing
> levels of fussiness. Then when the IETF starts shouting, there will be a
> short clean reference to accompany the shouting.
>
>
>
>
>
>
>
>