[Cbor] Re: dCBOR: Normalization of Strings

[Joe Hildebrand <[e-mail]>]

> On Jul 29, 2024, at 5:53 PM, Wolf McNally <[e-mail]> wrote:
> 
> Joe,
> 
> Thanks for your input.
> 
> My main problem with NFD is that it breaks existing Latin-1 characters with diacritics like `è` (0xe8) into two code points, requiring *three* bytes to encode as UTF-8:

Yes.

> • e (U+0065) encoded as 0x65 in UTF-8.
> • ̀ (U+0300) encoded as 0xCC 0x80 in UTF-8.
> 
> This means considerably more work and storage for very common use cases.

I think in most cases, you're unlikely to notice the difference, but of course it depends on your corpus.

> As I stated, Latin-1 (and by inclusion ASCII) is already in NFC by definition. So converting such a string to NFC would first detect whether it is already in Latin-1, and if so trivially avoid any further transformation: no decomposition then recomposition steps would be needed in these common cases.

I think that's true, assuming we're right that Latin-1 doesn't include any combining characters.  Whether taking two passes on non-Latin-1 strings is a net win probably also depends on your corpus.

Note: I don't really care which NF you pick or know whether you need to normalize for your use case.  I'm mostly pushing back on the IETF *always* recommending NFC.  The usual argument used to be that some font renderers couldn't handle decomposed characters -- that argument is no longer valid.

In a one XMPP server I worked on, NFKC processing of user IDs was *the* bottleneck, even after it had been optimized several times. The K part didn't make much difference in performance, but was of course destructive of information.  If we had it to do over, I would have argued hard for NFD.

-- 
Joe Hildebarnd