Re: [precis] Enforcement as an Idempotent operation

[Peter Saint-Andre <[e-mail]>]

Hi Bill, thanks for your message and sorry about the seriously delayed 
reply - I've been working to finish some other Internet-Drafts and now 
have time again to finish the PRECIS updates.

On 10/13/16 1:33 PM, William Fisher wrote:
> On Wed, Oct 12, 2016 at 9:03 PM, Peter Saint-Andre <[e-mail]> wrote:
>> It's not clear to me that U+1F11 has the problem you describe; perhaps could you sketch it out further?
>
> Oops, that should be U+0001F11A.

Did you mean U+212A (KELVIN SIGN)? That decomposes to U+004B (LATIN 
CAPITAL LETTER K).

> The full example is:
> "\U0001f11aevin" => "(K)evin" => "(k)evin"

Yes, "U+212Aevin" => "Kevin" via NFKC.

However, "U+212Aevin" => "kevin" via toLower() if I am not mistaken.

> I wrote a program to categorize characters that are not idempotent
> under Nickname "ToLower" (ignoring white space). The numbers are the
> same for Unicode 6.3, 8.0 and 9.0.
>
> {
>   '<font>': 467,
>   '<square>': 90,
>   '<compat>': 35,
>   '<super>': 27,
>   '<circle>': 4
> }

Would you mind sending me your list of characters? (I'm happy to receive 
it off-list.) I suspect that it might be similar to a list that emerged 
from differing assumptions regarding how to apply the PRECIS rules in 
implementations. My original implementation for testing purposes was 
rather naïve, whereas the implementation that Yoshiro Yoneya and 
Takahiro Nemoto created was smarter, in the sense that it would follow 
the chain of characters and decompose each one fully as it went along 
(this might require a few rounds of applying the normalization rule in 
order to fully decompose the original characters).

> The following two characters also appear to fail the idempotent test.
> The initial decompositions do not begin with '<'.
>
> \u03d3 GREEK UPSILON WITH ACUTE AND HOOK SYMBOL
> \u03d4 GREEK UPSILON WITH DIAERESIS AND HOOK SYMBOL

These examples are different from the KELVIN SIGN example, because there 
is no direct toLower() transformation - normalization needs to happen 
before toLower() is applied.

>> Thanks for your input. Personally I will think about it further and post again after I do so.
>
> To me, the problem is to take untrusted input, validate it using
> specified rules, and transform it into a stable, unambiguous format.
> I'm still learning more about Unicode.

Trust me: it never ends.

> Is there a reason that the case
> mapping rule has to be applied *before* the normalization rule?

As explained in Section 5.2.1 of RFC 7564, there is a good reason to 
apply the width mapping rule before the normalization rule. I'm now less 
sure that it makes sense, for comparison purposes, to apply the case 
mapping rule before the normalization rule.

> The
> order appears to make a difference for NFKC.  I suppose the Nickname
> "comparison" profile could re-apply the case mapping rule after the
> normalization rule?

If I understand correctly, you are suggesting that an implementation 
that is processing nickname strings for purposes of comparison would do 
the following:

1. Apply the "enforcement" action in Section 2.3
2. Apply the "comparison" action in Section 2.4

Let's choose a practical but somewhat contrived example: a nickname of 
ΨϓΧΗ, which is U+03A8 U+03D3 U+03A7 U+0397 (something like an uppercase 
version of the Greek word for soul, although the accent is wrong). This 
includes the code point U+03D3 that you mention above (which, by the 
way, is not the standard code point for the Greek letter upsilon but an 
alternative with a hook symbol, the usual character being U+03A5).

The two-step process you suggest would involve the following:

1. The "enforcement" action results in normalization (note that full 
normalization involves several steps):

U+03A8 U+03D3 U+03A7 U+0397
=>
U+03A8 U+03D2 U+0301 U+03A7 U+0397
=>
U+03A8 U+03A5 U+0301 U+03A7 U+0397

(note that U+03D2 has a compatibility equivalent of U+03A5)

2. The "comparison" action results in case mapping:

U+03A8 U+03A5 U+0301 U+03A7 U+0397
=>
U+03C8 U+03C5 U+0301 U+03C7 U+03B7

Thus, for comparison purposes, ΨϓΧΗ and ψύχη would be considered equivalent.

Unfortunately, even though that seems to yield the correct outcome, it's 
not what RFC 7700 specifies.

I'll continue to think about this - in particular, about any negative 
implications from modifying the order of operations so that 
normalization comes before case mapping (unlike what we specified in RFC 
7700). Because we would prefer that all the PRECIS specs follow the same 
order, we'd also need to look at the implications for RFC 7613 (although 
the OpaqueString profile that we use there for passwords has quite a 
different purpose than the Nickname profile in RFC 7700); however, this 
might not be possible.

Peter