Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

["Manger, James" <[e-mail]>]

Comments on draft-bray-unichars-03<https://www.ietf.org/archive/id/draft-bray-unichars-03.html>

Section 3.1. Unicode Code Points
The default repertoire of CBOR is unicode-scalar-values, not unicode-code-points. RFC8949 CBOR states that it’s string type “major type 3” is “a text string encoded as UTF-8”. That (since it is UTF-8) can’t include surrogates. It also states that “characters in this type are never escaped” so a JSON "\uDEAD" escape cannot be used to sneak in a surrogate. RFC8949 does use the phrase “Unicode code point” but appends “(scalar value)” at one point.

The default repertoire of JSON is not unicode-code-points since JSON excludes controls except tab, newline and carriage return. Given this spec distinguishes useful-assignables from unicode-scalar-values it should distinguish JSON’s actual subset from unicode-code-points if it is going to mention JSON.

3.1. needs to explicitly state that this unicode-code-points cannot be encoded in well-formed UTF-8 (or UTF-16 or UTF-32). It can only be used via higher-level escape sequences in protocols that offers those (such as JSON). This is mentioned in 2.2.1 (“it is impossible to represent a surrogate in well-formed UTF-8”), but also needs to be in 3.1. Otherwise, 3.1 and 3.2 appear as two similar choices, which elides their huge difference.

Section 2.2.3. Noncharacters
This spec highlights noncharacters for exclusion. However, Unicode explicitly warns against that: Corrigendum #9 Clarification About Noncharacters<https://www.unicode.org/versions/corrigendum9.html> says “the real intent of noncharacters is that they are permanently prohibited from being assigned standard, interchangeable meanings, rather than that they are prohibited from occurring in Unicode strings which happen to be interchanged”.
So an IETF spec is never going to define a string that needs a noncharacter; but it’s also never going to define a string that needs a private-use character either. If a spec defines an element that can hold any string, should that allow private-use characters but exclude noncharacters and non-useful controls? I’m not sure. That still leaves a lot of junk (eg BOM).

Section 5. Refining Character Repertoires
"\u7FFFF" is NOT a JSON escape for U+7FFFF; it a JSON escape for U+7FFF followed by an F character (as a few others have pointed out).
A proper JSON escape for U+7FFFF is "\uD9BF\uDFFF".

One form of escaping appearing in a character repertoire spec is jarring.

I agree that “many libraries will silently parse” "\uDEAD", but I’m not sure how many “generate an ill-formed UTF-8 string”. In Java, for instance, "\uDEAD".getBytes("UTF-8") returns a single byte 0x3F “?” – it’s valid UTF-8, just no longer a 1-to-1 representation of the in-memory unpaired surrogate.

--
James Manger




General

From: art <[e-mail]> on behalf of Tim Bray <[e-mail]>
Date: Saturday, 9 September 2023 at 5:38 am
To: [e-mail] <[e-mail]>, ART Area <[e-mail]>
Subject: [art] Just uploaded draft-bray-unichars-03
[External Email] This email was sent from outside the organisation – be cautious, particularly with links and attachments.
See https://www.ietf.org/archive/id/draft-bray-unichars-03.html

A bunch of minor corrections and improvements, thanks to everyone for that, especially James Manger for noticing that the ABNF was entirely wrong in one place.

The word “useless” has been replaced by “legacy”.

I think the feedback was pretty clear that the draft needed to be more opinionated; just because we document the existence of the default JSON repertoire (“all the code points”) doesn’t mean that anyone should use it in the present or future. So, introduced a new section “Refining Character Repertoires” to highlight those issues and offer a suggestion.