[Cbor] dCBOR: Normalization of Strings

[Wolf McNally <[e-mail]>]

All,

I’d like your comment on a proposed addition to dCBOR [1]:

• Encoders MUST only serialize text strings that are in Unicode Normalization Form C (NFC). [2]
• Decoders MUST require that deserialized text strings are in NFC.

This rule is comparable to dCBOR’s mandate for numerical reduction, as it applies to one of the most common used data types, and closes a frequently-discussed [3] determinism loophole regarding Unicode strings having more than one equivalent way to encode the same semantics. For example:

'e\u0301'  # 'e' followed by combining acute accent
'\u00E9'   # 'é' as a single character (the normalized form of this character)

These strings are canonically equal but encoded differently, breaking determinism if encoded by different agents without the proposed rule.

I have verified that there are readily-available ways to convert strings to NFC in C/C++ (using ICU or Boost libraries), Swift (using Foundation’s `precomposedStringWithCanonicalMapping`), Rust (using the `unicode-normalization` crate), JavaScript (using `String.prototype.normalize`), and PHP (using the `Normalizer` class). Several of these languages also have streamlined APIs for efficiently detecting whether a string is in NFC, which is useful for decoders. Those without such an API can simply compare a string being deserialized to a re-normalized copy.

I think it’s important to note that text strings in ASCII and Latin-1 are already necessarily in NFC, and the normalization rules are designed to be efficient in such common cases.

[1] dCBOR: A Deterministic CBOR Application Profile
https://datatracker.ietf.org/doc/draft-mcnally-deterministic-cbor/

[2] Unicode Standard Annex #15: Unicode Normalization Forms
https://unicode.org/reports/tr15/

[3] Discussion on Unicode normalization as it relates to canonical JSON.
https://esdiscuss.org/topic/json-canonicalize#content-3

~ Wolf