Re: [http-auth] Normalization forms in draft-ietf-httpauth-basicauth-enc

["Martin J. Dürst" <[e-mail]>]

On 2013/07/02 16:59, Yoav Nir wrote:
> Hi
>
> For those of us not so well versed in I18N issues, what do you mean by normalization?
>
> Is it just consolidating look-alike code points, like the multiple hyphens that exist in Unicode?
> Does it also involve removing Arabic and Hebrew points? (I think Paul raised this one)
> Does it involve removing diacritics?
> Does it involve splitting combined characters (like U+00E6 into 'a' and 'e')?
> Is there a standard we can point to and say "do this before comparing or hashing"?

In the narrow sense, it involves one of the four forms (NFC, NFKC, NFD, 
NFKD) defined in UAX #15 (http://www.unicode.org/reports/tr15/) NFC/D 
deal with precomposed/decomposed issues for accents, although not with 
cases such as U+00E6, where the distinction between "ae" and "æ" should 
be clearly visible. A Hebrew example would be U+FB1D, HEBREW LETTER YOD 
WITH HIRIQ, which as an exception is decomposed both in NFC and NFD to 
U+05D9 U+05B4 (usually, NFC is precomposed).

The K variants also include other variant changes, sometimes some very 
visible ones. For example, they take １ (full-width) and ① to 1. Here, 
most hyphen variants are not included, although some are (e.g. U+207B, 
subscript hyphen).

Case folding/equivalence is separate from both of the above cases.

Removing diacritics is also separate, because in many cases, that would 
be besides the point.

Paul has already mentioned stringprep, which is the base for nameprep 
and saslprep (and maybe others that I'm not aware of). That work is 
currently followed up by the work of the IETF precis WG.

Regards,   Martin.