Re: [I18nrp] Last Call: <draft-faltstrom-unicode11-05.txt> (IDNA2008 and Unicode 11.0.0) to Informational RFC

[Asmus Freytag <[e-mail]>]

On 12/4/2018 2:33 AM, John C Klensin wrote:
> Incidentally, if Asmus's comments about the insufficiency of the
> CONTEXTo model for a range of problems is correct, we should be
> looking at whether IDNA2008 is in need of changes/ updating.
> Those changes might turn out to be substantive or as minor as
> making it much more clear that what IDNA2008 permits is a
> superset of what should be allowed in a particular zone and who
> has responsibility (and accountability) for selecting a
> zone-specific subset (aided by whatever hints we can provide).
> It seems clear to me that we should not process this document
> until we have answers to, or at least much more clarity about,
> that issue.  YMMD.

Complex scripts (such as Indic and related scripts) are conceptually 
written as a stream of syllables. However, what is encoded is the set of 
elements that make up syllables. This leads to a well-formedness 
problem: it is possible to type sequences that are not well-formed 
syllables.

There are two kinds: syllables that "can't happen" given the phonetics 
of given language, and  syllables that are structurally unsound (violate 
the principles of the writing system).

Both can be problematic. The latter, because unsound sequences can look 
exactly like correct sequences, or, alternatively, because unsound 
sequences are not supported by renderers and fonts and lead to 
unpredictable display.

The "can't happen" sequences may cause issues to readers; the best 
analogy is that they are mentally processed like a nonsense letter (such 
as the ones invented by Dr. Seuss in "On Beyond Zebra"), instead of like 
a nonsense "word" as an arbitrary sequence of Latin letters would.

Structurally unsound sequences should be prohibited by policy, while 
some or all of the phonetically impossible sequences may be worth 
prohibiting for added security, but the latter are language dependent. 
(Some languages may differ in what constitutes a structurally sound 
syllable as well, an issue we encountered in the Thai script, where 
renderers designed for the Thai language will not handle some of the 
other languages).

Because of the (potential/actual) language dependency much of these 
constraints cannot be embedded in the protocol layer like IDNA 2008.

There are subset of unsound sequences that Unicode calls out as having a 
"do not use" status independent of language (See chapter 12 of the 
standard for examples). These could be captured with CONTEXTO (or 
similar) constraints in an updated IDNA2008.

However, they are always a subset of the "real" set of unsound 
sequences; any registry doing a good job for complex scripts would 
therefore find that implementing the CONTEXTO constraints becomes 
redundant. (Also, unlike RFC7940, these rules are not machine readable 
and therefore not trivial to implement and verify).

Fundamentally, IDNA2008 primarily focuses on repertoire, the few 
CONTEXTx rules notwithstanding. Changing that model would be a major 
undertaking and it is questionable that the results would be beneficial.

Perhaps some other tack could be taken: in your draft, it could be 
pointed out that Unicode assigns "do not use" to some sequences (which 
means that there is no text, identifier or not, that should contain 
them). Preventing these from registrations could be declared part of the 
responsibility of registries under RFC 5891.

This approach is more flexible because it allows registries to adopt 
context rules that are a superset of the rules needed to exclude these 
particular sequences. Which is precisely what we are doing in the Root 
Zone. If you go to http://icann.org/idn and look for the set of RZ-LGR 
proposals, you will find that the context rules for the various Indic 
scripts do not explicitly call out these "do not use" sequences, but 
that they adopt context rules on some of the constituent code points 
that also disallow those sequences, while implementing a more general 
constraint appropriate to the writing system.

For an example, see section 7 of 
https://www.icann.org/en/system/files/files/proposal-devanagari-lgr-27jul18-en.pdf

This shows the more generalized rules for Devanagari, but also where 
they had to be tweaked to allow for both Hindi and other languages, e.g. 
Santali.

For comparison, check for example, Table 12-1 in 
http://www.unicode.org/versions/latest/ch12.pdf

You will find that Table 12-1 is covered by the rule:

     3. M: must be preceded by C or CN

     (Read: "Dependent vowel signs (matras) must follow a consonant
      with or without an optional Nukta").

This rule prevents dependent vowel signs from being combined with 
independent vowel letters, which is what the list in table 12-1 
enumerates. However, the RZ-LGR rule also covers many other contexts 
that are inappropriate (unsound) for matras; some of the latter 
constraints may not be as universal, and, unlike the prohibition of 
Table 12-1 would be out of scope for a protocol.

The Root Zone LGR uses RFC 7940 to translate this rule into a single 
line of XML, which can be machine processed.

A./