Re: [Lucid] [[e-mail]: Re: Non-normalizable diacritics - new property]

[Ted Hardie <[e-mail]>]

On Wed, Mar 11, 2015 at 11:55 AM, Asmus Freytag (t) <[e-mail]
> wrote:

>  On 3/11/2015 10:58 AM, Ted Hardie wrote:
>
>  Hi Andrew,
>
>  So, I'm far from an expert on this, but I have some concerns about the
> proposal to use "visually" here in the way Mark proposes.  If I understand
> things correctly, this problem will also occur in any system that reads a
> sequence of characters aloud.  That is, there is no way to distinguish in
> an audio system between U+08A1 and the combination of U+0628 and U+0654.
>
>
> Where do get that?
>
> U+0654 is used to represent a glottal stop, but my understanding is that
> U+08A1 was encoded partially because it represents a sound that does not
> have a glottal stop. If my understanding is correct, the distinction should
> be quite audible.
>

I asked a native Arabic speaker about the systems she used and she played
them to check.  Those systems may be non-representative, though, or simply
wrong.  Or, rather than wrong, they may be tuned to a case in which the
result isn't audible.

Are we confident that this difference is general for the other cases?  If
so, that would be a very powerful way to explain the problem--code points
which are visually indistinguishable but would render differently in
audio.  (If it is not general, we still may want be cautious about focusing
solely on visual renderings).

regards,

Ted



>   The current draft talks about rendering the glyph, and I think that is
> closer to correct--it doesn't matter whether the rendering is audio or
> graphical for the problem to occur.  In the audio forms, the  quality of
> running text providing context vs. independent identifiers without it also
> seems similar, at least to me.
>
>
> I think that the focus on visual as suggested by Mark ignores that there
> are identifiers for which the user community claims that certain visually
> distinct representations are considered "the same" by users, to the point
> that one cannot rely on users to faithfully retain which version of the
> identifier is the desired one.
>
> As the problem statement is concerned with general background, it is
> important to recognize such other levels of ambiguity exist.
>
> A./
>
>
>  I also don't believe that Mark's point about  parsing of byte sequences
> as XY when X and Y or meant is quite correct; in string contexts that would
> be a parsing error.  It might not be easily distinguished on the wire, but
> in protocol slots, this seems to be an at best unrelated issue.  I may not
> be familiar, though, with contexts in which this may occur.
>
>  regards,
>
>  Ted
>
> On Tue, Mar 10, 2015 at 6:33 PM, Andrew Sullivan <[e-mail]>
> wrote:
>
>> Dear colleagues,
>>
>> I hereby forward a message sent to a few people (including Asmus and
>> me) responding to the draft Asmus and I submitted yesterday.
>>
>> I think this is a valuable response, though I am not sure I agree with
>> the approach it's taking.  Nevertheless, if we are to have any hope of
>> useful discussion in Dallas, it'd be good to have a look at the issues
>> in advance.
>>
>> Best regards,
>>
>> A
>>
>> ----- Forwarded message from Mark Davis ?️ <[e-mail]> -----
>>
>> Date: Tue, 10 Mar 2015 16:53:33 +0100
>> From: Mark Davis ?️ <[e-mail]>
>> To: "Asmus Freytag (t)" <[e-mail]>
>> Cc: Roozbeh Pournader <[e-mail]>, Ken Whistler <
>> [e-mail]>, Lisa
>>         Moore <[e-mail]>, Michel Suignard <[e-mail]>,
>> Markus
>>         Scherer <[e-mail]>, Peter Constable <
>> [e-mail]>,
>>         [e-mail]
>> Subject: Re: Non-normalizable diacritics - new property
>>
>> + Andrew
>>
>> ​> ​
>> In the meantime, the following has been released in preparation to the BOF
>> at the IETF meeting in Dallas.
>>
>> ​> ​
>> https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/
>> <https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/>
>>
>>
>> A well written report, and I think will help a great deal in resolving the
>> issues.
>>
>> However, I'd strongly suggest a couple of changes to be more precise, and
>> thus avoid confusion (no pun intended). Although these may seem overly
>> formal, it is vital that the text be unambiguous, because the subject is
>> so
>> very tricky, and it is thus so easy for people to be arguing based on
>> different interpretations of terms.
>>
>> I will try to suggest some minimal changes, although it might be even
>> better with some more extensive rewording.
>>
>> > I3 depends on the assumption that strings that will be used in
>> >   identifiers will not have any ambiguous matching to other strings.
>>
>> When discussing strings, the term "ambiguous matching" is itself quite
>> ambiguous. I can have ambiguous matching with SJIS, for example, because
>> 0x61 can be an 'a' or be part of another character; similarly there are
>> sequences of bytes ..XY.. where XY is a character or X is the end of one
>> character, and Y is the start of another. There are other forms of
>> ambiguity as well.
>>
>> To eliminate this ambiguity, change every 8 other cases of "ambiguous/ity"
>> to prefix by "visually".
>>
>> =>   identifiers will not be visually ambiguous with other strings used as
>> identifiers.
>> ...
>>
>> Look for other cases that could profit by that, such as like
>> "indistinguishable", and "matches".
>>
>> See the later:
>>
>> > Worse, identifiers, by their very nature, are things that must
>> >   provide reliable exact matches.  The whole point of an identifier is
>> >   that it provides a reliable way of uniquely naming the thing to be
>> >   identified.
>>
>> In this section, to be precise, one would need to use "visual matches".
>> There are so, so many other ways strings can "match", like "have the same
>> code points".
>>
>> However, the second sentence is still problematic. The characters in two
>> different strings that have similar appearances *are* different
>> characters,
>> so the identifiers are "unique" in that sense. I guess what you meant to
>> say is something like:
>>
>> => it provides a reliable way of naming the thing to be identified, so as
>> to have a unique visual appearance.
>>
>> Now, I think adding "visual" makes it clear that this paragraph needs some
>> work. As it stands, it would be too strong: it would call for eliminating
>> all homographs from all identifiers (eg disallowing "top" written in
>> Cyrillic characters.)
>>
>> Later, there is a shift to confusability, but the text never relates that
>> directly to "confusability", which is what is elaborated upon.
>>
>> (In general, many different phrases are (apparently) used to mean the same
>> thing: matching, indistinguishable, etc. While use of different terms
>> makes
>> the text flow better, it has the big disadvantage that the reader never
>> knows whether phrase1 is meant to have a broader or narrower scope than
>> phrase2, or is meant to be identical. This would be a more extensive
>> change, however, to use more consistent defined language. Alternatively,
>> at
>> the top you could say that when you use the terms "matches",
>> "indistinguishable", "ambiguous", etc. (you'll have to go through and find
>> them all) that what is meant is in terms of visual appearance.)
>>
>> > (We use the term "homoglyph" strictly: code points that normally use the
>> same glyph when rendered.)
>>
>> This needs a bit of work. First step would be:
>>
>> Strings S1 and S2 are strict homoglyphs when they are different, yet
>> normally use the same glyph sequence when rendered.
>>
>> Second would be to clarify "normally use". Does that mean "in the fonts
>> that most people use on most platforms"? There are often more significant
>> differences in serif fonts than non-serif, for example. Does that require
>> that the same font is used? There is huge variation among glyphs in
>> fonts: ţ
>>  and ț may look the identical in font 1, and may be discernibly different
>> when both are in font2, but ţ in font1 may also look identical to ț in
>> font2. This also gets tricky because in many systems there is font
>> fallback. If a character is not in font1, then it might be displayed in a
>> fallback font 2.
>>
>> Third would be to clarify "same glyph sequence". Does this mean pixel by
>> pixel? At all resolutions, or just some? Pixel by pixel is extremely
>> strict.
>>
>> One could try to bring in "intent", but that is very dicy. The intent is
>> from the Unicode encoding side may not be followed precisely by all or
>> even
>> most font vendors. So it is of only theoretical value to point to intent.
>>
>> The exact degree of strictness matters a great deal, because under a
>> sufficiently strict interpretation <ARABIC LETTER BEH (U+0628) + ARABIC
>> HAMZA ABOVE (U+0654)> is *not* a strict homograph for U+08A1, ARABIC
>> LETTER
>> BEH WITH HAMZA ABOVE. That is, most current fonts do not display them as
>> pixel-for-pixel identical.
>>
>> I know well that this is a continuum, but since you are focusing on strict
>> homoglyphs, you have to be clearer what you mean by that term; where you
>> draw the line. That also carries over into related terms elsewhere in the
>> document like "the same glyph"; does that mean in all fonts (or
>> some?/most?//common?), pixel for pixel or not, at all sizes or just
>> body-text sizes, etc.?
>>
>> You might think about providing a definition for "the same glyph
>> sequence",
>> then using that in the definition for homograph, and elsewhere in the
>> text.
>>
>> > Mitigation may be as simple as using a font
>>    designed to distinguish among different characters.
>>
>> Should mention that in practice this is extremely difficult to enforce; it
>> only works in closed systems, those that have complete control over the
>> fonts used for display of the kinds of identifiers in question.
>>
>> ----- End forwarded message -----
>>
>> --
>> Andrew Sullivan
>> [e-mail]
>>
>> _______________________________________________
>> Lucid mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/lucid
>>
>
>
>
> _______________________________________________
> Lucid mailing [e-mail]://www.ietf.org/mailman/listinfo/lucid
>
>
>