Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

[Steffen Nurpmeso <[e-mail]>]

Manger, James wrote in
 <[e-mail]\
 ook.com>:
 |[1]draft-bray-unichars[/1] §3 “Dealing with problematic code points” \
 |suggests “replacing problematic code points with "�" (U+FFFD, REPLACEMENT \
 |CHARACTER)” (or signalling an error, 
 |but I’ll only talk about the replacement option in this email).
 |
 |  [1] https://datatracker.ietf.org/doc/html/draft-bray-unichars
 |
 |* An ill-formed sequence of code units needs to be replaced. It is \
 |far less obvious to me that “problematic” scalars should be replaced. \
 |Even for noncharacters Unicode provides a 
 |good [2] FAQ[/2] and [3]corrigendum #9 “Clarification about noncharact\
 |ers”[/3] that suggests passing them along (treating them like unassigned \
 |scalars) is often the best policy 
 |(because the internal/interchange boundary is blurry).
 |So §4.3 defining unicode-assignable that excludes noncharacters is \
 |fine -- when to be lenient on receiving a supposed unicode-assignable \
 |value is less obvious.
 |But §3 looks dodgy.
 |
 |* U+FFFD is an obvious choice to replace code units or scalars you \
 |don’t want. But Unicode does allow choices. [4]Unicode ch3[/4] C10 \
 |only says “with a marker such as U+FFFD”. [5] 
 |Unicode TR36[/5] says “where U+FFFD is not available, a common alternative \
 |is "?"”. Java, for instance, uses “?” is some common circumstances. \
 |Unichars does not admit such an option.

The situation on the iconv(3) (POSIX, Portable Operating System
Interface) front is a disaster really.  Let me quote some code
snippet of the honourable Bruno Haible of GNU iconv and more:

  /* Irix iconv() inserts a NUL byte if it cannot convert.
     NetBSD iconv() inserts a question mark if it cannot convert.

("Citrus" that is, also used on some other BSD's.)
The small (but very neat, and lots of "impressive" code snippets)
musl C library as used for example by AlpineLinux uses asterisk *.

     Only GNU libiconv and GNU libc are known to prefer to fail rather
     than doing a lossy conversion.  */

Irix made an incredible bad choice.
Anyhow it is all totally intransparent for programmers anyhow,
since you do know nothing of at least one character set, neither
the "resolved name", nor whether it is multi-byte or "multi-word"
or "multi-multi-{word,multi}", not whether it has "state", not
whether ASCII NUL can be a vivid part (of "unused" "word"s),
nothing.

Off-topic but i am still hoping we get a dedicated _addressable_
behaviour switch for iconv(3) that then does _not_ overload the
EILSEQ error (as there is no "invalid input", just missing output
convertability).

Of course, this is all about boundaries in between different
character sets with non-Unicode involved.

Just to add that the most widely used conversion library GNU iconv
does for many years:

      UTF-8: Reject surrogates and out-of-range code points.
      * lib/utf8.h (utf8_mbtowc, utf8_wctomb): Reject code points in the
      range 0xD800..0xDFFF and >= 0x110000.

As in, "IETF is one thing, but programmers have to deal with that
on the code side".

No no.  UTF-8 as in RFC 3629, and with _exactly_ the "well-formed"
check as defined by

      if(LIKELY(x <= 0x7Fu))
              c = x;
      /* 0xF8, but Unicode guarantees maximum of 0x10FFFFu -> F4 8F BF BF.
       * Unicode 9.0, 3.9, UTF-8, Table 3-7. Well-Formed UTF-8 Byte Sequences */
      else if(LIKELY(x > 0xC0u && x <= 0xF4u)){
      ...
      }else
              goto jerr;

I am a bit outdated.

If some _sender_ pushes through garbage because it has to, and
i have had requests to disable iconv so that garbage is sent over
as mail for the MUA i maintain, mind you, there are administrators
and they want the mail report to be sent out no matter what
garbage there is!, .. and let me tell it was requested by
a Dr. aka Bachelor++, even a Master-of-the-Universe, these are the
real life guys, heaven!, and, furthermore, that over fourty years
old protocol with its almost thirty years MIME extensions actually
CAN deliver this s..t!!, then, if the IETF is about to define that
network protocols should use UTF-8, then it should define that the
one and only Unicode character which was specifically designed for
that purpose -- to indicate that some non-representable had to be
represented -- should be used.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)