Re: [I18nrp] next steps

[Nico Williams <[e-mail]>]

On Wed, Jul 25, 2018 at 06:54:27PM -0700, Larry Masinter wrote:
> Here's a wild idea, flame away
> 
> > [...]
> 
> It isn't clear how Unicode defines "normalization" compared to how the
> term is used in mathematics. If you start with an equivalence relation

The important thing is that Unicode defines canonical normalization
forms.  I don't believe there is an argument that those forms aren't
normal -- that is, that they allow multiple equivalent forms in one NF,
which clearly they do not.

> https://en.wikipedia.org/wiki/Equivalence_relation 
> you can define a "canonicalization" if the set has a total ordering,
> and pick the "least" or "most". But the form you might get by doing so
> might not be desirable, or suitable to be called "normal", even if
> it's useful for determining equivalence.
> 
> You don't need to normalize locally for using hash-table caching
> if you allow the hash-table to reflect a cached redirect,
> and put the onus on the origin to supply a canonical string.

Take as an example LATIN LETTER U WITH DIAERESIS AND MACRON U+0304.
This character can be expressed in this many ways:

  - U+1E7B
  - U+0075 U+0308 U+0304
  - U+0075 U+0304 U+0308
  - U+016B U+0308

That's four different ways, only two of which are in one NF or another.

Now imagine a string that has four such characters.  You'd not have 4^4
possible combinations -- 256(!!) of them only two (or maybe four) of
which would be in one NF or another (well, four, if any of those
characters have different K forms).

I'm really not familiar enough with input methods to know how many of
all those combinations are actually likely in the real world, but
because most input methods pre-compose, and because HFS+ decomposes,
you'll have at least two forms.  Any more than two forms and this can't
scale, and it saves you nothing.

Better to normalize as you hash (and only if necessary -- all-ASCII
strings, for example, require no normalization) then.

> You might think this would bloat the table with all of the
> possible redirects, but as long as you look up on data entry

Not just bloat the table.  Also bloat compute, and if you have to have
these sent to you, then bloat network bandwidth.

> and the origin is consistent, there should only be one redirect.
> In fact, you could limit it to one, or just periodically clear LRU.

The form-insensitive string comparison algorithm I described is actually
quite fast in most cases, as it can manage to avoid normalization in
most cases.  Normalizing as you hash is not as fast, but it's OK.

> I'd even include case folding, punycode and %xx-URL percent encodings
> as possible input forms. The times they are typed in are
> rare.

If you're hash table allows duplicates, sure, then you can still be
case-sensitive.  Otherwise only if you want case-insensitivity.

> It would allow different TLDs to define their own additional
> equivalences and case folding.
> 
> Some variation of this could be made to work for DNS, HTTP,
> and email addresses with some enormous deployment problems.
> Would need to prove that it works.

IMO it would not work at all.

> Sorry if this is terse

I got it.

I don't push form-insensitivity only as an optimization.  IMO it has the
best, most user-friendly semantics and interoperability characteristics,
and it minimizes the number of locations where normalization
capabilities are needed.  That's at least three big reasons to prefer
f-i.

Nico
--