Re: [art] [I18ndir] Modern Network Unicode

[John C Klensin <[e-mail]>]

--On Thursday, July 11, 2019 18:44 +0200 Carsten Bormann
<[e-mail]> wrote:

> On Jul 11, 2019, at 18:31, John C Klensin <[e-mail]>
> wrote:
>> 
>> (top post and adding the ART list back in)
>> 
>> Carsten, 
>> I think Asmus and I are in complete agreement, but you might
>> find an attempt to summarize the state of work for relevant
>> characteristics and behaviors on the web, some of which
>> interact with this discussion, illuminating.  It is a chart
>> with explanation, not a long document.  See
>> https://w3c.github.io/typography/gap-analysis/language-matrix
>> .html 
> 
> Thank you for this interesting link.
> This is very condensed information; what column should I look
> for specifically on the issue of normalization and
> applicability of NFC? E.g., in the linked page
> https://w3c.github.io/iip/gap-analysis/beng-gap I find
> something about grapheme clusters but nothing about
> normalization — normalization of course is not an immediate
> need for Web interoperability.

I'll try to hunt for it when I have time unless someone else
gets to it first, but there are W3C recommendations that
specifically recommend against trying to insert a normalization
step before transmission or immediately before storage but
focusing on it as a comparison-time operation.   That is
consistent with what Asmus and I have been trying to tell you.  

Let me try to summarize those comments in a different way:

(1) For most language-script combinations, the form in which a
network application gets a string is usually going to be the
right form for that string.  If there is a strong preference
that the string be normalized, it will probably come to the
network application (and probably when it leaves something
rather close to the keyboard) normalized.  If there is a strong
preference that it not be normalized, it will probably come to
you in _that_ form.  If there is no preference at all,
normalizing the string (as long as you stay away from NFKC/NFDC)
will probably not cause harm, but it won't buy much either.

(2) Checking as to whether a string is normalized before sending
it is unlikely to produce any safely actionable information
unless you know a lot about the language, script, and
circumstances.

(3) If a string is unnormalized for what whomever created it
thinks is a good reason, then normalizing on general principles
is going to lose whatever information that decision was intended
to convey.  

The above may be different for strings that are specifically
intended as identifiers, but I don't think that is what you are
talking about.    I used to believe that "normalize everywhere"
and "store strings only in normalized form" was a good idea, but
I've been gradually (and sometimes quite unpleasantly) educated
by a long series of cases and counterexamples.   If I were
updating 5198 today, that recommendation is the thing I would be
most likely to either change or at least moderate and qualify
considerably.

    john