Re: [apps-discuss] i18n intro, Sunday 14:00-16:00

["Martin J. Dürst" <[e-mail]>]

Hello Joe,

On 2011/07/22 1:28, Joe Hildebrand wrote:
> On 7/21/11 1:03 AM, "Martin J. Dürst"<[e-mail]>  wrote:
>
>> Slide 123: Good to see that. By the way, I seem to remember both John
> and me
>> begging you for an explanation of why Jabber wants to use NFD a
> few months
>> ago, and I'm not sure I have seen an answer. Now might be a
> good time (if you
>> already sent one, a pointer would be appreciated).
>
>
> Let me try.

Glad to start talking.

> First some assumptions:
> - Stringprep is currently one of the performance hotspots of some XMPP
> servers.

Is that an assumption backed by facts or a wild guess?

> - XMPP does not guarantee that the original form of the address that is
> entered by the user or sent on the first hop is transmitted without
> modification to other hops in the system.
> - As such, many XMPP servers optimize by performing canonicalization at the
> edges of their system and even store the canonical version for future
> comparison.

Makes sense.

> - If the spec is written that clients SHOULD perform canonicalization, many
> in our community will, particularly if they know that they will get better
> performance from the server.

That's the same for NFC and NFD, isn't it? The advantage of NFC is that 
it's designed to more-or-less match what's out there, so you get the 
advantage that there is more stuff that's already canonicalized even if 
if a client doesn't do anything. That gets reinforced by both the IETF 
and the W3C telling everybody to use NFC.

> The property of NFK?D that we like is that if you have a string of
> codepoints that is already in NFK?D, you can check that the string is in the
> correct normalization form without having to allocate memory.  With NFK?C,
> you'll have to decompose (allocating memory), recompose (at some finite CPU
> cost), then recompose (possibly allocating *again*) just to check if you
> have already done the normalization.

Nope. That's just how the algorithm was defined, because the 
decomposition component was already around, and NFD was what defined 
canonical equivalence.

As an example, http://www.w3.org/2003/06/xml1.1test/Overview.html has 
some code (both UTF-8 and UTF-16) for compact (memory footprint) and 
reasonably fast NFC check. Please ignore the "XML 1.1" in the title. 
Also please note that this is proof of concept code, and may need 
additional testing and of course upgrade to the newest version of 
Unicode. I don't have any open bug reports, but that may be for other 
reasons than that there are no bugs.

Also, there are various ways to trade off speed against memory (see also 
Björn's mail), but the memory here is just the footprint of the sharable 
data, there's no need for lots of memory per conversion.


> For the K portion, I have found John's argument compelling that codepoints
> with compatibility decompositions should just be prohibited in our
> localparts.

Probably just fine for most cases. Potentially a problem for wide/narrow 
in places like Japan.

> In our resourceparts, I'm of the opinion that we don't need to
> compatibility map -- it's fine for all of those codepoints to stay distinct.

Yes indeed.

> The idea is that clients SHOULD normalize, servers double-check inputs from
> non-trusted sources (like clients and other servers), then always store and
> forward the normalized version.

That's fine. But it doesn't explain the choice of NFD (vs. NFC).

Regards,   Martin.