Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-04.txt

["Manger, James" <[e-mail]>]

I don’t think draft-bray-unichars-04<https://www.ietf.org/archive/id/draft-bray-unichars-04.html> should advance in the IETF.

Defining unicode-scalar-values, xml-chars, and useful-assignables are 3 helpful “subsets of Unicode characters” that can be used in protocols and data formats.
Defining unicode-code-points as though it is similar is a category error, however.

Section 3.1 and 3.2 deliberately make %x0-10FFFF and %x0-D7FF / %xE000-10FFFF looks like similar repertoires, which is too misleading.
UTF-8, UTF-16 and UTF-32 are only defined for the latter. There may be “obvious” extensions of UTF-8 (WTF-8) and UTF-32 that can cover %x0-10FFFF, but they are simply not widely supported in modern software, so those extensions are no use in an IETF standard. And the “obvious” extension to UTF-16 gives something that no longer works as you expect from a repertoire.

An implementation that accepts surrogates cannot distinguish a {high-surrogate, low-surrogate} pair from a non-BMP character. ECMA-404 is clear on this when it says, “whether a processor of JSON texts interprets such a surrogate pair (“\uD834\uDD1E”) as a single code point (U+1D11E) or as an explicit surrogate pair is a semantic decision that is determined by the specific processor”. That is totally unexpected from seeing %x0-10FFFF as a seemingly simple repertoire.

It makes sense for a spec to define:
  unicode-scalar-value = %x0-D7FF / %xE000-10FFFF
  string = *unicode-scalar-value

It does not make sense for a spec to define:
  unicode-code-point = %x0-10FFFF
  string = *unicode-code-point
because, for instance, %xD834 %xDD1E and %x1D11E are separate values of that ABNF grammar but will not be treated that way by implementations. In implementations they will be indistinguishable strings. An internal 16-bit format will store the same two 16-bit words for both. Only 1 form can come out.

For understandable reasons, JSON supports both *(%x0-D7FF / %xE000-10FFFF) and *(%x0-FFFF) (arbitrary 16-bit data) as models for the logical strings it can represent. An implementation can pick either. They don’t exactly overlap. There is probably a complicated ABNF that can cover both involving *(%x0-D7FF / %xE000-10FFFF)  and unpaired-surrogate, but it would be non-trivial (and not that practical). And that ABNF for the logical strings JSON can represent is different from the ABNF for JSON text itself, which excludes controls other than whitespace and has escape sequences. *(%x0-10FFFF) – as implied by 3.1 – doesn’t match any concept here.

The Unicode spec (chapter 2 General Structure<https://www.unicode.org/versions/Unicode15.0.0/ch02.pdf>) covers abstract characters, code points, codespace, code unit, encoding form, encoding scheme etc. draft-bray-unichars-04 is shorter and simpler, but the simplifications aren’t quite precise enough to add clarity.

P.S. The ABNF productions in draft-bray-unichars-04 should be singular not plural, eg unicode-scalar-value.

--
James Manger



General