Re: [Lucid] [[e-mail]: Re: Non-normalizable diacritics - new property]

[John C Klensin <[e-mail]>]

Ted,

I won't claim to more than about 10% more expertise, but maybe I
can clarify some of the issues you raise (and with which I
generally agree)...

--On Wednesday, March 11, 2015 10:58 -0700 Ted Hardie
<[e-mail]> wrote:

> Hi Andrew,
> 
> So, I'm far from an expert on this, but I have some concerns
> about the proposal to use "visually" here in the way Mark
> proposes.  If I understand things correctly, this problem will
> also occur in any system that reads a sequence of characters
> aloud.

Or not.  To give more familiar example than the one that started
this (I'm assuming there are no first-language speakers of Fula
on this list and, if there are, that they probably don't
primarily write it in Arabic characters), text to speech
programs generally need to know the language involved and often
have language-specific dictionaries.  Anyone who speaks
competent French knows that English vowel phonemes and French
ones are different.  When one set is used to render the other
language, what first-language speakers of the second language
hear is "weird accent" at best and speech that is
incomprehensible at worst.   English is particularly bad in this
regard because it has a lot fewer vowel (and consonant, but the
vowels are more obvious) symbols than it has phonemes, while
many languages that use Latin Script "decorate" their characters
in various ways that provide much more written distinction among
those phonemes.

In written form, type style and rendering distinctions, which
are very much tied up with "visual", involve many of the same
issues because accurate rendering is often tied up with
language, rather than being intrinsic properties of the script.

Interestingly, as soon as a suggestion is made that takes us to
the "language" distinctions that are often key to both "visual"
and "spoken", it also brings up back to where the original
discussion about U+08A1 --the discussion that led to our
understanding of the extent and complexity of these problems--
started: if one knows the language contexts, the distinctions
that Unicode makes by not having some characters decompose
become obvious.  Conversely, if one it talking about the DNS or
other identifiers in which language information is inherently
unavailable or non-applicable, then statements and ideas that
depend on language just don't help.

There are a second set of distinctions that get involved here
and that do not depend on language -- a collection of combining
characters that (using a summary with which Unicode experts
might not agree) are not defined precisely enough that one can
predict how they will overlay on or combine with the base
characters.  The resulting precomposed characters (almost)never
decomponse.  This issue is discussed briefly in the forth
paragraph of Section 3.3.1 of
draft-klensin-idna-5892upd-unicode70-04 and the piece of the
Unicode Standard referenced there as "Unicod70-Overlay" (sic - I
hate it when a typo is spotted immediately after a document is
posted) so this isn't _just_ a language-distinction (or
phoneme-distinction) problem, but that is part of it.

>  That is, there is no way to distinguish in an audio
> system between U+08A1 and the combination of U+0628 and
> U+0654.

Actually, if we are willing to drag the discussion back into
into Hamza issues (one of the reasons why
draft-klensin-idna-5892upd-unicode70-04 is twice the length of
draft-sullivan-lucid-prob-stmt), there is a significant phonetic
difference.  See section 2.2.2 of the latter and a different
(and more comprehensive) version in 3.2.3 of the former and, as
needed, the Unicode explanations it references.  I prefer to
think of these distinctions as language ones rather than
phonetic ones because if they were the latter, no normal human
being would be able to tell which code point to use even though
reading (i.e., text to speech) systems would be happier.  If
both the combining sequence and the precomposed character could
reasonably appear with the same language and be treated as
distinct, there would be a big problem but, if that problem
exists, I haven't seen an example yet.

The other problem with "visual" is that it pulls us back into
subjective confusability and that is both a nightmare we would
prefer to avoid if possible and something that has been the
source of earlier claims that there is no solvable problem in
this area.

>  The current draft talks about rendering the glyph,
> and I think that is closer to correct--it doesn't matter
> whether the rendering is audio or graphical for the problem to
> occur.  In the audio forms, the  quality of  running text
> providing context vs. independent identifiers without it also
> seems similar, at least to me.
>...

    john