[Lucid] [[e-mail]: Re: Non-normalizable diacritics - new property]

[Andrew Sullivan <[e-mail]>]

Dear colleagues,

I hereby forward a message sent to a few people (including Asmus and
me) responding to the draft Asmus and I submitted yesterday.

I think this is a valuable response, though I am not sure I agree with
the approach it's taking.  Nevertheless, if we are to have any hope of
useful discussion in Dallas, it'd be good to have a look at the issues
in advance.

Best regards,

A

----- Forwarded message from Mark Davis ?️ <[e-mail]> -----

Date: Tue, 10 Mar 2015 16:53:33 +0100
From: Mark Davis ?️ <[e-mail]>
To: "Asmus Freytag (t)" <[e-mail]>
Cc: Roozbeh Pournader <[e-mail]>, Ken Whistler <[e-mail]>, Lisa
	Moore <[e-mail]>, Michel Suignard <[e-mail]>, Markus
	Scherer <[e-mail]>, Peter Constable <[e-mail]>,
	[e-mail]
Subject: Re: Non-normalizable diacritics - new property

+ Andrew

​> ​
In the meantime, the following has been released in preparation to the BOF
at the IETF meeting in Dallas.

​> ​
https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/
<https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/>


A well written report, and I think will help a great deal in resolving the
issues.

However, I'd strongly suggest a couple of changes to be more precise, and
thus avoid confusion (no pun intended). Although these may seem overly
formal, it is vital that the text be unambiguous, because the subject is so
very tricky, and it is thus so easy for people to be arguing based on
different interpretations of terms.

I will try to suggest some minimal changes, although it might be even
better with some more extensive rewording.

> I3 depends on the assumption that strings that will be used in
>   identifiers will not have any ambiguous matching to other strings.

When discussing strings, the term "ambiguous matching" is itself quite
ambiguous. I can have ambiguous matching with SJIS, for example, because
0x61 can be an 'a' or be part of another character; similarly there are
sequences of bytes ..XY.. where XY is a character or X is the end of one
character, and Y is the start of another. There are other forms of
ambiguity as well.

To eliminate this ambiguity, change every 8 other cases of "ambiguous/ity"
to prefix by "visually".

=>   identifiers will not be visually ambiguous with other strings used as
identifiers.
...

Look for other cases that could profit by that, such as like
"indistinguishable", and "matches".

See the later:

> Worse, identifiers, by their very nature, are things that must
>   provide reliable exact matches.  The whole point of an identifier is
>   that it provides a reliable way of uniquely naming the thing to be
>   identified.

In this section, to be precise, one would need to use "visual matches".
There are so, so many other ways strings can "match", like "have the same
code points".

However, the second sentence is still problematic. The characters in two
different strings that have similar appearances *are* different characters,
so the identifiers are "unique" in that sense. I guess what you meant to
say is something like:

=> it provides a reliable way of naming the thing to be identified, so as
to have a unique visual appearance.

Now, I think adding "visual" makes it clear that this paragraph needs some
work. As it stands, it would be too strong: it would call for eliminating
all homographs from all identifiers (eg disallowing "top" written in
Cyrillic characters.)

Later, there is a shift to confusability, but the text never relates that
directly to "confusability", which is what is elaborated upon.

(In general, many different phrases are (apparently) used to mean the same
thing: matching, indistinguishable, etc. While use of different terms makes
the text flow better, it has the big disadvantage that the reader never
knows whether phrase1 is meant to have a broader or narrower scope than
phrase2, or is meant to be identical. This would be a more extensive
change, however, to use more consistent defined language. Alternatively, at
the top you could say that when you use the terms "matches",
"indistinguishable", "ambiguous", etc. (you'll have to go through and find
them all) that what is meant is in terms of visual appearance.)

> (We use the term "homoglyph" strictly: code points that normally use the
same glyph when rendered.)

This needs a bit of work. First step would be:

Strings S1 and S2 are strict homoglyphs when they are different, yet
normally use the same glyph sequence when rendered.

Second would be to clarify "normally use". Does that mean "in the fonts
that most people use on most platforms"? There are often more significant
differences in serif fonts than non-serif, for example. Does that require
that the same font is used? There is huge variation among glyphs in fonts: ţ
 and ț may look the identical in font 1, and may be discernibly different
when both are in font2, but ţ in font1 may also look identical to ț in
font2. This also gets tricky because in many systems there is font
fallback. If a character is not in font1, then it might be displayed in a
fallback font 2.

Third would be to clarify "same glyph sequence". Does this mean pixel by
pixel? At all resolutions, or just some? Pixel by pixel is extremely strict.

One could try to bring in "intent", but that is very dicy. The intent is
from the Unicode encoding side may not be followed precisely by all or even
most font vendors. So it is of only theoretical value to point to intent.

The exact degree of strictness matters a great deal, because under a
sufficiently strict interpretation <ARABIC LETTER BEH (U+0628) + ARABIC
HAMZA ABOVE (U+0654)> is *not* a strict homograph for U+08A1, ARABIC LETTER
BEH WITH HAMZA ABOVE. That is, most current fonts do not display them as
pixel-for-pixel identical.

I know well that this is a continuum, but since you are focusing on strict
homoglyphs, you have to be clearer what you mean by that term; where you
draw the line. That also carries over into related terms elsewhere in the
document like "the same glyph"; does that mean in all fonts (or
some?/most?//common?), pixel for pixel or not, at all sizes or just
body-text sizes, etc.?

You might think about providing a definition for "the same glyph sequence",
then using that in the definition for homograph, and elsewhere in the text.

> Mitigation may be as simple as using a font
   designed to distinguish among different characters.

Should mention that in practice this is extremely difficult to enforce; it
only works in closed systems, those that have complete control over the
fonts used for display of the kinds of identifiers in question.

----- End forwarded message -----

-- 
Andrew Sullivan
[e-mail]