Re: [I18nrp] next steps

[Nico Williams <[e-mail]>]

On Mon, Jul 23, 2018 at 08:38:15PM -0700, Asmus Freytag wrote:
> If you define equivalence relations (e.g. variants) based on code points,
> then you'll need to apply these / compare strings in NFD otherwise you will
> miss out on the effect of re-ordering of combining sequences.

Or NFC.  It doesn't matter which form you choose as the internal form
for comparison, provided that the comparison is form-insensitive.

Now, since normalization to NFC is defined in terms of an initial
normalization to NFD, it's obviously best to use NFD internally anyways.

Note, BTW, that form-insensitive comparison is not necessarily enough to
implement form-insensitivity.  If you have hash tables, say, and you're
hashing strings for hash table lookups, then you have to normalize
during hashing as well in order to have form-insensitive behavior.

The nice thing about this form-insensitive comparison/hashing is that it
can be done with fixed and small memory allocation, as you can step
strings character by character, and there is a maximum length in
codepoints to a character normalized to NFD, thus the amount of memory
needed is fixed, and that memory can be allocated on the stack (where
you have one).

Indeed, you can even optimize form-insensitive string comparison so that
for the mostly-ASCII and mostly-not-equivalent or mostly-equal input
cases... most characters do not require canonical decomposition.

Nico
--