Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-06.txt

["Manger, James" <[e-mail]>]

Why is U+D800 so much more crucial than U+20FFFF or an ill-formed UTF-8 byte sequence such as 0xC0 0x80?

There are lots of byte sequences that are ill-formed UTF-8. Some correspond to non-minimal-length encodings (eg C0 80 ~ U+0000); some to encoding surrogates (eg ED A0 80); some to encoding values greater than U+10FFFF (eg F7 BF BF BF ~ U+1FFFFF); some that just don’t fit UTF-8’s structure (eg FF). A robust system needs to cope with all of these (signalling an error or using a marker such as U+FFFD). draft-bray-unichars only covers surrogates. That’s poor. I feel it adds to the confusion about what is lurking in Unicode by conflating surrogates with abstract characters, whereas it is much clearer to group surrogates with other ill-formed code unit sequences.

> the explanation of the problem works better if you start from code points

I disagree. Problems include:

  1.  Arbitrary code unit sequences can be ill-formed
  2.  BOM
  3.  Noncharacters
  4.  Legacy controls
  5.  Tag chars and other deprecated format chars
  6.  NFC vs NFKC; canonical chars
  7.  Private use chars
  8.  BIDI
  9.  …
  10. Escaping

#1 is about encodings. #2 is about encodings and is a char. #3-7 are about char subsets. #10 is about a higher-level.
Code points don’t help explain any of these.

> motivate why scalars exist

Explaining the 1,081,344 size and the U+D800-U+DFFF gap would be interesting. I’d love to read about why that gap was chosen and not, say, U+F800-U+FFFF. My guess is that that was an unused (or little used) part of 16-bit charsets at the time. Scalars could have been defined as 0-1081343, but I guess that would mean more math when converting 16-bit code units to & from scalars. Stick it in an appendix, however, as it doesn’t help the docs primary purpose.

> I don’t think the discussion about scalars and UTF-8/16/32 is appropriate. UTF-16 and UTF-32 shouldn’t be used in protocols, the Internet has converged on UTF-8

Argh!
Sure, only use UTF-8 in protocols. But there is also memory & APIs where 16-bit code units (hence UTF-16) are still important. A set to work across UTF-8/16/32 is the motivation for scalars. And this doc only started because JSON uses 16-bit code units in its escape mechanism.
If we are only interested in UTF-8 why are surrogates ever mentioned?

--
James Manger




General

From: Tim Bray <[e-mail]>
Date: Sunday, 1 October 2023 at 8:46 am
To: Manger, James <[e-mail]>
Cc: [e-mail] <[e-mail]>, ART Area <[e-mail]>
Subject: Re: [art] Fwd: New Version Notification for draft-bray-unichars-06.txt
[External Email] This email was sent from outside the organisation – be cautious, particularly with links and attachments.
This is interesting, thank you James.

James proposes a restructure that cleverly removes mentions of code points and surrogates, on the basis that scalars are more important than code points.

He also points out an important error; the I-JSON repertoire is not scalars, it’s scalars minus noncharacters. I had entirely forgotten this.  Should we introduce an I-JSON subset for people who might want to use control codes?  I personally think using control codes is a bad idea, do others disagree?

While I agree that scalars are more important than code points, I have a problem with some of this, which is not technical.  I think the most common scenario in which this document would be used (should it progress), would be someone who is working on an internet draft with text fields and gets told “you need to specify your character repertoire, go look at [Unichars]”.

So the document exists at least in part to educate people about what’s lurking in Unicode.   For that reason, I don’t want to have a spec in the style of “just do this, don’t worry about why, trust us”. I think the explanation of the problem works better if you start from code points and explicitly caution against surrogates. Also, it’s hard to motivate why scalars exist without introducing surrogates.

I don’t think the discussion about scalars and UTF-8/16/32 is appropriate. UTF-16 and UTF-32 shouldn’t be used in protocols, the Internet has converged on UTF-8, RFC2277 mandates it, and this document should assume that characters on the wire are UTF-8.

What do people think about the BOM?  Officially, it’s ZERO WIDTH NO-BREAK SPACE.  It probably doesn't add value to a UTF-8 field but it also probably won’t break anything.  I can see people using it to pad out fixed-length fields?  If we remove it from “Unicode Assignables” then we probably have to include an explanation about the BOM functionality.



On Sep 30, 2023 at 2:15:01 AM, "Manger, James" <[e-mail]<mailto:[e-mail]>> wrote:
Comments on draft-bray-unichars-06<https://datatracker.ietf.org/doc/html/draft-bray-unichars>.

§2 “Characters and Code Points” should start with scalars; they are far more important than code points. Suggested replacement text.

[UNICODE<http://www.unicode.org/versions/latest/>] defines the 1,081,344 integers in the ranges 0 to D7FF16 and E00016 to 10FFFF16 as "Unicode scalars". Every character is assigned to one scalar. As of Unicode 15.1 (2023), 149,813 characters have been assigned, leaving 931,531 scalars available for assignment in future versions.

unicode-scalar = %x0-D7FF / %xE000-10FFFF

Scalars are the complete set of values that can be uniquely represented in all 3 Unicode encoding forms – UTF-8, UTF-16, and UTF-32 – which use 8-bit, 16-bit and 32-bit code units respectively. So scalars are the repertoire that works for representing characters in memory, storage, and in network protocols regardless of choices to use 8, 16 or 32-bit words.

I’d rename the section to be “Characters and scalars”.

Drop §2.1 “Transformation formats”.

§2.2 “Problematic Code Point Types” could be renamed “Problematic characters”.

BOM should be considered a problematic character (and excluded from unicode-assignables) as it is used as an encoding-layer signal.

Surrogates are handled at the encoding layer, not the character layer, so drop §2.2.1 “Surrogates”. I suggest a new §2.3 “Ill-formed encodings” to replace §2.2.1 and most of §3.

2.3. Ill-formed encodings

A sequence of 8-bit, 16-bit or 32-bit code units representing scalars is a well-formed UTF-8, UTF-16, or UTF-32 encoding respectively. However, there are other code unit sequences in each of these 3 encodings that don’t map to scalars (eg C016 8016 in UTF-8; D80016 in UTF-16; 20FFFF16 in UTF-32). Such sequences are call ill-formed. They can exist in practice. Reasonable options when interpreting such code unit sequences are signalling an error or treating them as "�" (U+FFFD, REPLACEMENT CHARACTER). Silently ignoring ill-formed code unit sequences is a known security risk.

Drop §3 “Dealing With Problematic Code Points”.

Typo: \U0089 should be \u0089.

Typo: RFC19413 should be RFC9413.

I’d define unicode-scalar in §2 so we don’t need §4.1 “Unicode Scalars”. §4 can say:

Specifications can refer to these by the names “Unicode scalars” (section 2), “XML Characters”, and “Unicode Assignables”.

I-JSON can’t be used as an example using unicode-scalar as it explicitly excludes noncharacters; and the difference between the repertoire for the JSON vs the repertoire for the logical string that can be represented by a JSON string is not explained.

i-json-value-repertoire = %x9 / %xA / %xD / %x20-D7FF / %xE000-FFFD / %x10000-1FFFD / %x20000-2FFFD / … / %x100000-10FFFD

i-json-logical-string-repertoire = %x0-D7FF / %xE000-FFFD / %x10000-1FFFD / %x20000-2FFFD / … / %x100000-10FFFD
--
James Manger




General
From: art <[e-mail]<mailto:[e-mail]>> on behalf of Tim Bray <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, 26 September 2023 at 2:51 am
To: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>, ART Area <[e-mail]<mailto:[e-mail]>>
Subject: [art] Fwd: New Version Notification for draft-bray-unichars-06.txt
[External Email] This email was sent from outside the organisation – be cautious, particularly with links and attachments.
What’s new and different here.


  1.  Locked down definition of “problematic”
  2.  Locked down definition of “character repertoire”
  3.  Changed “Useful Assignables” to “Unicode Assignables” (checked with Asmus first)

A new version of Internet-Draft draft-bray-unichars-06.txt has been
successfully submitted by Paul Hoffman and posted to the
IETF repository.

Name:     draft-bray-unichars
Revision: 06
Title:    Unicode Character Repertoire Subsets
Date:     2023-09-25
Group:    Individual Submission
Pages:    10
URL:      https://www.ietf.org/archive/id/draft-bray-unichars-06.txt
Status:   https://datatracker.ietf.org/doc/draft-bray-unichars/
HTML:     https://www.ietf.org/archive/id/draft-bray-unichars-06.html
HTMLized: https://datatracker.ietf.org/doc/html/draft-bray-unichars
Diff:     https://author-tools.ietf.org/iddiff?url2=draft-bray-unichars-06

Abstract:

  This document discusses specifying subsets of the Unicode character
  repertoire for use in protocols and data formats.



The IETF Secretariat