Re: [I18n-discuss] Comments on "troublesome-characters" from Arabic script

[John C Klensin <[e-mail]>]

Andrew, Dr. Al-Zoman,

Let me suggest what may be a middle ground (a variation of one
I've already suggested off-list to Andrew and Asmus) to see if
it works for you.   I think the key here is an expansion of
things Andrew has already said, which is that this particular
draft should not (in IETF-speak, MUST NOT) be interpreted out of
context with other documents.  Partially for that reason, it may
need additional text to explain those connections.

FWYW, my hypothesis is that Arabic script is not the only one
that would be adversely affected if "troublesome" were
interpreted as "bad character" or "don't allow this".  We were
just lucky that you and your colleagues spotted the problems
early, mentioned it, and were able to clearly explain the issue.

Would you agree with the following:

(1) The use of combining characters with Arabic script is
problematic, to the point of requiring special attention and
knowledge whether they should be prohibited or not.  AFAICT,
combining characters are not allowed by the analysis and
recommendations of RFC 5564 and, by adding more precombined
forms, Unicode has decided to not encourage their use.

(2) Arabic script as defined by Unicode is unique in having two
sets of code points for digits within the same script.  The
observation that European digits are often used with the Arabic
language in some places further complicates that problem.

(3) At least for the part of the world's population that does
not use them every day, scripts that are normally written with
the characters connected (a category that is certainly not
limited to Arabic) are more of a problem than scripts written
with more obvious character boundaries.  This does not reflect
badly on those scripts in any way, but it does make it harder
for those who are ignorant about the scripts to look characters
up in tables, transcribe written forms into computer input, etc.

(4) Again, at least for those who do not use them every day,
scripts that are normally written right to left, especially
those that also use numerals that are written left to right
(i.e., with the highest-valued digit on the left), are more
difficult than left to right ones.  While this is not a
significant problem with the scripts themselves, it can, unless
care is exercise by people who understand what is going on,
result in strange behavior in some circumstances.
Fully-qualified domain names where some labels start or end with
digits have proven to be particularly difficult examples. 

Now, if we agree on that much, I suggest that the document
should probably say the following (in different words) and say
it very clearly and strongly:

(i) Arabic script (because of the first two characteristics
above) and _any_ script with either or both of the other two
characteristics make the requirements that a registry understand
the script(s) it allows, that it carefully design a plan for
working with each of those scripts, and that it consider itself
accountable for the results, even more important than those
requirements usually are.  Code points that are, or are not,
listed by this document may be useful as input to such registry
understanding and policy development efforts, but are not a
substitute for them.  Again, those principles apply to all
registries and all scripts, but the ones identified by (1)-(4)
above require extra attention (so, by the way, do registries
allowing more than one of Greek, Latin, or Cyrillic scripts and
several other combinations -- perhaps the document should
include a section of "troublesome" combinations of scripts).

(ii) The rule of numeral homogeneity (RFC 5564, Section 2.3.1),
expanded to also prohibit mixing with "Eastern" Arabic-Indic
digits, is far more important and useful than trying to call out
any of those digit code points as problematic.  

(iii) Combining marks are extremely problematic with Arabic
script.  If combining marks are prohibited by the registry, then
code points that are listed in the table as problematic because
of possible combining sequences become non-problematic.  While,
if I understand RFC 5564 and your comments correctly, simply
prohibiting all of them is reasonable for the Arabic language, I
note that the latest version of the ASIWG recommendations that I
can find (June 2008,
http://arabic-domains.org/docs/jun2010/IDNA200x_Arabic_Script_Disallowed_Codepoints.pdf)
[1] allows a large number of combining characters.

(iv) If recommendations for domain name use (or identifier use
more generally) have been developed by groups reflecting a broad
range of expertise, including experts on the relevant
linguistics and writing systems (not just, e.g.,
name-marketing), for a particular script or language and those
recommendations are very specific about what they cover and are
applied carefully, they are most likely to be a better guide to
what is appropriate and/or troublesome than a list, including
this "troublesome" one, developed for general use.  For the
Arabic language written in Arabic script and a domain registry
that does not allow labels based on other languages written in
Arabic script, that implies that RFC 5564 is a better guide than
this document, no matter what this one says.  By contrast, if
one had a registry that was exclusively registering labels based
on, e.g., Persian writing conventions, this document (perhaps in
combination with the generic recommendations for Arabic script
in the ASIWG work) would be more appropriate than trying to
apply 5564 (if specific and carefully-developed recommendations
for Persian existed, that would, of course, be better yet).  

Given an explanation of that type, the vast majority of the
Arabic code points could either be removed from the Internet
Draft or turned into cross-references to the explanation or more
authoritative documents.

Would that be a reasonable direction to take?

best regards,
    john

[1] Were the ASIWG recommendations ever completed and formally
approved by ESCWA and the relevant countries?   The table
mentioned above is the latest I can find but it is dated even
earlier than the 2009 meeting I attended, and the asiwg.org site
to which several references point appears to now be in Chinese
script.  An up-to-date reference, if it exists, would be
appreciated.

--On Sunday, July 30, 2017 08:37 +0000 "Abdulaziz H. Al-Zoman"
<[e-mail]> wrote:

> Good day Andrew,
> 
> Thanks for your follow-up email to my feedback.
> 
> 
> I do understand the goal of the internet draft as you've
> stated:      "to create the conditions for guidance
>       for operators and applications, so that
>       it is possible for (for instance) my
>       user agent to work globally, even when
>       some parts of the global linguistic
>       environment is foreign to me and when
>       there are no in-protocol clues about
>       the language I'm facing."
> 
> but I'm worried by the inclusion of "basic" code points (22
> out of 28 Arabic language alphabet) to the repository instead
> of only the problematic code points (non-spacing marks). This
> would make the Arabic language unusable (or troublesome) to
> operators and developers. While, I would like to see some
> encouragement to support Arabic IDNs rather than shun them
> away from it.
> 
> Therefore,  I would suggest that the registry includes only
> the problematic sequence of code points that incorporate
> non-spacing marks but not the basic (and essential)
> characters. 
> 
> So I would suggest that the repository table looks like the
> following table (for example) where: Column 1 represent the
> problematic (sequence of) code points and Column 2 contains
> the Reasons and Comments. The Column 1 should NOT contain a
> basic code point alone without non-spacing mark causing the
> problem.
> 
> Column 1             | Column 2
> ---------------------+------------------
> 062F, 065C           | Identical in appearance to ...
> ---------------------+------------------
> 062F, 06EC           | Identical in appearance to ...
> ---------------------+------------------
> 0631, 06EC           | Identical in appearance to ...
> ---------------------+------------------
> 0633, 06DB           | Identical in appearance to ...
> 
> 
> Yours,
> Abdulaziz Al-Zoman
>