[Cbor] Re: dCBOR: Normalization of Strings

[Wolf McNally <[e-mail]>]

Joe,

> On Jul 29, 2024, at 3:18 PM, Joe Hildebrand <[e-mail]> wrote:
> 
> I think that's true, assuming we're right that Latin-1 doesn't include any combining characters.

It definitely does not.

The Unicode Standard Annex #15 “Unicode Normalization Forms” also states:

> Note: Text exclusively containing ASCII characters (U+0000..U+007F) is left unaffected by all of the Normalization Forms. This is particularly important for programming languages. (See Unicode Standard Annex #31, "Unicode Identifier and Pattern Syntax" [UAX31].) Text exclusively containing Latin-1 characters (U+0000..U+00FF) is left unaffected by NFC. This is effectively the same as saying that all Latin-1 text is *already* normalized to NFC.
https://unicode.org/reports/tr15/

> Whether taking two passes on non-Latin-1 strings is a net win probably also depends on your corpus.

I don’t think you need two passes, and I highly doubt NFC converters do. It only requires a single pass: for each character ask, “is it Latin-1”. If so, copy it to the output. IFF you discover a non Latin-1 character, then do the decomposition and recomposition steps. If you want to avoid allocation for the trivial case, then do this pass with the test, and IFF you find a non-Latin-1 character then copy the result so far to the output and proceed with the more complex case. If you never find a non-Latin-1 character, return the original input.

> That would be a reasonable choice as well, depending on your use case.  Would you expect the decoder to fail decoding if it received a non-Latin-1 character?  Or would that be some other layer's job?

As it is the dCBOR decoder’s job to enforce dCBOR compliance, a *fully* compliant decoder would fully follow the rule. Failing that, a less-than-fully compliant decoder SHOULD do what it can to enforce the rules, but SHOULD also document that it is not fully-compliant. In the case of Latin-1, ensuring the code points are in 0 <= p <= 255 is sufficient:

```python
def isAllLatin1(utf8_string):
    try:
        for char in utf8_string:
            if ord(char) > 255:
                return False
        return True
    except UnicodeDecodeError:
        return False
```

This check would never admit a non-NFC string, but could also give false negatives on NFC strings that aren’t Latin-1.

If your environment was so constrained that you needed to drop compliance for this rule entirely, then you could just validate it as UTF-8 and then accept it.

```python
def is_valid_utf8(data):
    try:
        data.decode('utf-8')
        return True
    except UnicodeDecodeError:
        return False
```

If you needed to drop even that level of compliance, then you'd basically just treat the string like a blob and hope for the best.

~ Wolf