Re: [I18ndir] Review of new characters for Unicode 12.0.0

[Martin J. Dürst <[e-mail]>]

Hello Asmus,

Many thanks for the followup, also on the Unicode side.

On 2019/03/18 13:12, Asmus Freytag wrote:
> On 3/17/2019 5:49 PM, Martin J. Dürst wrote:
>> There were some talks about doing a review of the new characters for
>> Unicode 12.0.0, for 'due diligence'.
>>
>> Here are the results such a review of new characters for Unicode 12.0.0,
>> starting off http://www.unicode.org/charts/PDF/Unicode-12.0/.
>> I also used a small Ruby program that I wrote, attached. In order for it
>> to work, you have to make sure you use the latest (and greatest :-)
>> version of Ruby that supports Unicode 12.0.0.
>>
> ....
> 
> 
>> Latin Extended-D
>> http://www.unicode.org/charts/PDF/Unicode-12.0/U120-A720.pdf
>> 4 casing pairs, 3 upper-case letters completing case pairs with existing
>> lower-case letters. Upper-case are disallowed (-> okay), lower-case are
>> pvalid. Among these 4, the following three may merit a closer look:
>> U+A7BB LATIN SMALL LETTER GLOTTAL A
>> U+A7BD LATIN SMALL LETTER GLOTTAL I
>> U+A7BF LATIN SMALL LETTER GLOTTAL U
>> They are graphically very similar (if not identical) to the following
>> three from the Latin extensions for Vietnamese:
>> (see https://www.unicode.org/charts/PDF/U1E00.pdf)
>> U+1EA3 ả LATIN SMALL LETTER A WITH HOOK ABOVE
>> U+1EC9 ỉ LATIN SMALL LETTER I WITH HOOK ABOVE
>> U+1EE7 ủ LATIN SMALL LETTER U WITH HOOK ABOVE
>> I have sent a mail to Unicode experts to get more information on these.
> 
> 
> For A7BD (and it's uppercase, which is not relevant for IDNs) the 
> documents say:
> 
> "The code positions could be U+A7BA and U+A7BB (next available code 
> positions), properties as follow:
> A7BA;LATIN CAPITAL LETTER EGYPTOLOGICAL YOD;Lu;0;L;;;;;N;;;;A7BB;
> A7BB;LATIN SMALL LETTER EGYPTOLOGICAL YOD;Ll;0;L;;;;;N;;;A7BA;;A7BA
> There are not confusable with any existing code points,

After close examination, I indeed realized that the above Vietnamese 
characters have hooks (half-circles with a small downwards stroke 
attached at the lower end) whereas the 'glottal' characters have 
something more like U+2019 ’ RIGHT SINGLE QUOTATION MARK. So at least on 
close inspection, they are distinguishable. Making sure that diacritic 
marks are easily distinguishable in a requirement of any font that's 
claiming to be suited for IDNA anyway, so this isn't new territory.
[I remember the famous font designer Chuck Bigelow talking about this in 
a keynote at an Unicode conference many many years ago, see also
http://cajun.cs.nott.ac.uk/compsci/epo/papers/volume6/issue3/bigelow.pdf, 
section entitled "The design of diacritics".]


> but the new 
> characters could be confused with the sequence of the Latin letter I, 
> Latin letter IOTA, Greek letter IOTA, followed by a diacritical mark 
> resembling the top of the glyph. Given the existing confused situation, 
> this does not exacerbate the risk in a meaningful way. There is a 
> limited need to use a YOD in general purpose identifiers; its main 
> purpose is to represent transliteration of historic Egyptian text 
> originally written using Egyptian hieroglyphs." (WG2 N4792R2)

For others, this is at 
https://unicode.org/wg2/docs/n4792r2-egyptologicalYod.pdf.

> On this basis, they could be exceptionally DISALLOWED for IDNA.
> 
> The other two characters are for transcriptions of Ugaritic - also 
> something not desperately needed and exceptional "disallowed" could be 
> defended.

Yes. The risk is then that some as-yet unalphabetized language community 
picks up some of these letters, and later finds out they cannot use them 
in IDNs.

Anyway, I'd be open to any kind of conclusion if it can be made quickly 
(weeks rather than years).

Given they are indeed distinct from the 'Vietnamese' characters, I'd be 
okay with just going with the result of the algorithm (i.e. PVALID), but 
they might be candidates for your 'problematic characters' document.

> See WG2 document N3487.

That's http://www.unicode.org/L2/L2008/08272-n3487-egyptological-yod.pdf.

Regards,    Martin.