Re: [http-auth] Normalization forms in draft-ietf-httpauth-basicauth-enc

On Jul 2, 2013, at 4:18 PM, Peter Saint-Andre <stpeter@stpeter.im> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 7/2/13 1:59 AM, Yoav Nir wrote:
>> Hi
>> 
>> For those of us not so well versed in I18N issues, what do you mean
>> by normalization?
> 
> Martin described it already.
> 
> You might find it useful to look at the slides I put together for a
> two-hour tutorial on internationalization at an IETF meeting a few
> years ago:
> 
> https://stpeter.im/files/i18n-into.pdf
> 
> Perhaps I need to offer that tutorial again? Or maybe turn it into a
> small book?

Excellent resource. Yeah, you should.

>> Is it just consolidating look-alike code points, like the multiple
>> hyphens that exist in Unicode?
> 
> There is no complete solution to the problem of confusable characters.
> Here is a fun example for you:
> 
> https://stpeter.im/journal/1420.html
> 
> However, normalization takes care of some (but not all) instances of
> code points that look alike.

For our particular use case we need the results of normalization to be bit-for-bit equal, and to get the same result when typing the name or password in different browsers on PCs, Macs, all Linux distros, and all smartphones and tablets. As mentioned earlier in this thread, these are not always identical.

>> Does it also involve removing Arabic and Hebrew points? (I think
>> Paul raised this one)
> 
> There is no reason to *remove* characters from left-to-right scripts.
> However, allowing them means you need to deal with interesting bidi
> issues.

There is. In French, André is correct while Andre is simply wrong. In Arabic and Hebrew, the diacritics are reading aids that were bolted on to an existing script. The text is fine with and without them, so I could use either יואב or יוֹאָב for my name, and both would be perfectly fine and in some sense equivalent. It's still fine to decide that these forms are not equivalent for our purposes, so if you used points for your username when you created it, you'll have to use points forever.

>> Does it involve removing diacritics?
> 
> Not necessarily -- why can't my username be saintandré or whatever?
> And *removing* diacritics might be problematic, if it implies that 'é'
> would be transformed to 'e' (thus introducing the possibility of
> additional false positives).

It all goes back to the keyboard the user uses to input username and password. I have no idea how to type the example above on Windows or on any phone. I just don't know the key combinations (yes, I could google it). So having your username be saintandré could leave you unable to log in from some platforms. The case could be made that we need stronger normalization than other use cases.

>> Does it involve splitting combined characters (like U+00E6 into 'a'
>> and 'e')?
> 
> Actually, there is no decomposition of U+00E6 æ into 'a' and 'e' in
> Unicode. This might be counter-intuitive, but it's true.
> 
>> Is there a standard we can point to and say "do this before
>> comparing or hashing"?
> 
> I'd suggest looking carefully at SASLprep (RFC 4013) and its proposed
> replacement:
> 
> http://datatracker.ietf.org/doc/draft-ietf-precis-saslprepbis/
> 
> If you have feedback on the latter, please post to the precis@ietf.org
> list:
> 
> https://www.ietf.org/mailman/listinfo/precis
> 
> Peter