Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-04.txt

[Tim Bray <[e-mail]>]

 OK, I think your core point is that the current structure of the draft
could lead someone to pick U+0000-U+10FFFF as a reasonable character
repertoire.  Neither Paul nor I read it that way, but you’re not crazy and
if it reads that way to not-crazy people, then we should fix it.

We still have to describe U+0000-U+10FFFF, if only because if you specify
the use of JSON and don’t specify a subset repertoire, that’s what you’re
going to get, as for example the people who implement AWS services know,
and have to explicitly defend themselves.  But if we stop calling it a
subset and restructure the document, we can probably address your concern
without pretending that U+0000-U+10FFFF doesn’t exist in the wild.

On Sep 18, 2023 at 7:05:05 AM, "Manger, James" <
[e-mail]> wrote:

> I don’t think draft-bray-unichars-04
> <https://www.ietf.org/archive/id/draft-bray-unichars-04.html> should
> advance in the IETF.
>
>
>
> Defining unicode-scalar-values, xml-chars, and useful-assignables are 3
> helpful “subsets of Unicode characters” that can be used in protocols and
> data formats.
>
> Defining unicode-code-points as though it is similar is a category error,
> however.
>
>
>
> Section 3.1 and 3.2 deliberately make %x0-10FFFF and %x0-D7FF /
> %xE000-10FFFF looks like similar repertoires, which is too misleading.
>
> UTF-8, UTF-16 and UTF-32 are only defined for the latter. There may be
> “obvious” extensions of UTF-8 (WTF-8) and UTF-32 that can cover %x0-10FFFF,
> but they are simply not widely supported in modern software, so those
> extensions are no use in an IETF standard. And the “obvious” extension to
> UTF-16 gives something that no longer works as you expect from a repertoire.
>
>
>
> An implementation that accepts surrogates cannot distinguish a
> {high-surrogate, low-surrogate} pair from a non-BMP character. ECMA-404 is
> clear on this when it says, “whether a processor of JSON texts interprets
> such a surrogate pair (“\uD834\uDD1E”) as a single code point (U+1D11E) or
> as an explicit surrogate pair is a semantic decision that is determined by
> the specific processor”. That is totally unexpected from seeing %x0-10FFFF
> as a seemingly simple repertoire.
>
>
>
> It makes sense for a spec to define:
>
>   unicode-scalar-value = %x0-D7FF / %xE000-10FFFF
>
>   string = *unicode-scalar-value
>
>
>
> It does not make sense for a spec to define:
>
>   unicode-code-point = %x0-10FFFF
>
>   string = *unicode-code-point
>
> because, for instance, %xD834 %xDD1E and %x1D11E are separate values of
> that ABNF grammar but will not be treated that way by implementations. In
> implementations they will be indistinguishable strings. An internal 16-bit
> format will store the same two 16-bit words for both. Only 1 form can come
> out.
>
>
>
> For understandable reasons, JSON supports both *(%x0-D7FF / %xE000-10FFFF)
> and *(%x0-FFFF) (arbitrary 16-bit data) as models for the logical strings
> it can represent. An implementation can pick either. They don’t exactly
> overlap. There is probably a complicated ABNF that can cover both involving
> *(%x0-D7FF / %xE000-10FFFF)  and unpaired-surrogate, but it would be
> non-trivial (and not that practical). And that ABNF for the logical strings
> JSON can represent is different from the ABNF for JSON text itself, which
> excludes controls other than whitespace and has escape sequences.
> *(%x0-10FFFF) – as implied by 3.1 – doesn’t match any concept here.
>
>
>
> The Unicode spec (chapter 2 General Structure
> <https://www.unicode.org/versions/Unicode15.0.0/ch02.pdf>) covers
> abstract characters, code points, codespace, code unit, encoding form,
> encoding scheme etc. draft-bray-unichars-04 is shorter and simpler, but the
> simplifications aren’t quite precise enough to add clarity.
>
>
>
> P.S. The ABNF productions in draft-bray-unichars-04 should be singular not
> plural, eg unicode-scalar-value.
>
>
>
> --
>
> James Manger
>
>
>
> General
>