[Cbor] Re: dCBOR: Normalization of Strings

[Carsten Bormann <[e-mail]>]

On 29. Jul 2024, at 18:00, Joe Hildebrand <[e-mail]> wrote:
> 
> Why would text in your application be in NFC form already?

I’m thinking about applications that build CBOR data items from data that are in a database.
While database normalization is much more than what we are discussing here, the database could make sure to have NFC forms of the text strings it stores.

> Note that you have to at least check that whatever strings that needed to be normalized is in the correct format before trusting that the deterministic encoding was done correctly, which means every step in the process that relies on determinism is going to be doing this expensive check.

This is a good point — one may be able to identify steps that actually do not need that verification.

> I'd need to know more about the particular application to have an opinion on whether all of the text needs to be normalized.

The text will be used for signing inputs, or hash values will be extracted that simply will not match if the normalization isn’t done.

>  User-entered text that doesn't need to be compared against strings that were entered somewhere else probably don't need normalization, even for deterministic encoding.

Right, but that assumes you keep this text as entered, as opposed to canonical forms in a database.
(In many cases, you’ll need functions such as search that does provide access to a normalized form.)

Grüße, Carsten