Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-06.txt

["Manger, James" <[e-mail]>]

Comments on draft-bray-unichars-06<https://datatracker.ietf.org/doc/html/draft-bray-unichars>.

§2 “Characters and Code Points” should start with scalars; they are far more important than code points. Suggested replacement text.

[UNICODE<http://www.unicode.org/versions/latest/>] defines the 1,081,344 integers in the ranges 0 to D7FF16 and E00016 to 10FFFF16 as "Unicode scalars". Every character is assigned to one scalar. As of Unicode 15.1 (2023), 149,813 characters have been assigned, leaving 931,531 scalars available for assignment in future versions.

unicode-scalar = %x0-D7FF / %xE000-10FFFF

Scalars are the complete set of values that can be uniquely represented in all 3 Unicode encoding forms – UTF-8, UTF-16, and UTF-32 – which use 8-bit, 16-bit and 32-bit code units respectively. So scalars are the repertoire that works for representing characters in memory, storage, and in network protocols regardless of choices to use 8, 16 or 32-bit words.

I’d rename the section to be “Characters and scalars”.

Drop §2.1 “Transformation formats”.

§2.2 “Problematic Code Point Types” could be renamed “Problematic characters”.

BOM should be considered a problematic character (and excluded from unicode-assignables) as it is used as an encoding-layer signal.

Surrogates are handled at the encoding layer, not the character layer, so drop §2.2.1 “Surrogates”. I suggest a new §2.3 “Ill-formed encodings” to replace §2.2.1 and most of §3.

2.3. Ill-formed encodings

A sequence of 8-bit, 16-bit or 32-bit code units representing scalars is a well-formed UTF-8, UTF-16, or UTF-32 encoding respectively. However, there are other code unit sequences in each of these 3 encodings that don’t map to scalars (eg C016 8016 in UTF-8; D80016 in UTF-16; 20FFFF16 in UTF-32). Such sequences are call ill-formed. They can exist in practice. Reasonable options when interpreting such code unit sequences are signalling an error or treating them as "�" (U+FFFD, REPLACEMENT CHARACTER). Silently ignoring ill-formed code unit sequences is a known security risk.

Drop §3 “Dealing With Problematic Code Points”.

Typo: \U0089 should be \u0089.

Typo: RFC19413 should be RFC9413.

I’d define unicode-scalar in §2 so we don’t need §4.1 “Unicode Scalars”. §4 can say:

Specifications can refer to these by the names “Unicode scalars” (section 2), “XML Characters”, and “Unicode Assignables”.

I-JSON can’t be used as an example using unicode-scalar as it explicitly excludes noncharacters; and the difference between the repertoire for the JSON vs the repertoire for the logical string that can be represented by a JSON string is not explained.

i-json-value-repertoire = %x9 / %xA / %xD / %x20-D7FF / %xE000-FFFD / %x10000-1FFFD / %x20000-2FFFD / … / %x100000-10FFFD

i-json-logical-string-repertoire = %x0-D7FF / %xE000-FFFD / %x10000-1FFFD / %x20000-2FFFD / … / %x100000-10FFFD
--
James Manger




General

From: art <[e-mail]> on behalf of Tim Bray <[e-mail]>
Date: Tuesday, 26 September 2023 at 2:51 am
To: [e-mail] <[e-mail]>, ART Area <[e-mail]>
Subject: [art] Fwd: New Version Notification for draft-bray-unichars-06.txt
[External Email] This email was sent from outside the organisation – be cautious, particularly with links and attachments.
What’s new and different here.


  1.  Locked down definition of “problematic”
  2.  Locked down definition of “character repertoire”
  3.  Changed “Useful Assignables” to “Unicode Assignables” (checked with Asmus first)

A new version of Internet-Draft draft-bray-unichars-06.txt has been
successfully submitted by Paul Hoffman and posted to the
IETF repository.

Name:     draft-bray-unichars
Revision: 06
Title:    Unicode Character Repertoire Subsets
Date:     2023-09-25
Group:    Individual Submission
Pages:    10
URL:      https://www.ietf.org/archive/id/draft-bray-unichars-06.txt
Status:   https://datatracker.ietf.org/doc/draft-bray-unichars/
HTML:     https://www.ietf.org/archive/id/draft-bray-unichars-06.html
HTMLized: https://datatracker.ietf.org/doc/html/draft-bray-unichars
Diff:     https://author-tools.ietf.org/iddiff?url2=draft-bray-unichars-06

Abstract:

  This document discusses specifying subsets of the Unicode character
  repertoire for use in protocols and data formats.



The IETF Secretariat