Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-04.txt

["Manger, James" <[e-mail]>]

TR17 and Unicode are careful about the distinction between scalars, code points, code units. Scalars and code units are far and away the most important. draft-bray-unichars-04 tries to simplify, but by emphasising code points it adds confusion & inaccuracy.

Plenty of systems will parse the JSON "\uDEAD" to & from an internal string representation, but that JSON is 8 ASCII characters (including the quotes) – there is nothing ill-formed there. It’s far less clear to me if many systems will parse "<ill-formed code unit sequence for U+DEAD>" from five 8-bit WTF-8 code units, three WTF-16 code units, or three WTF-32 code units. And it’s unclear to me whether ECMA-404 expects anything to parse those.

Let’s consider the ABNF
  str = *(%x0-10FFFF)
and two values
  str1 = %x1D11E
  str2 = %xD834 %xDD1E
Both str1 and str2 are valid str values. They are different (1 symbol vs 2). Any ABNF tool will say they are different. If internally my system uses 32-bit code units these will be different: arrays of 1 and 2 code units respectively. But somehow if my system uses 16-bit code units internally then str1 and str2 are identical – indistinguishable. If my system uses 8-bit code units str1 will be four bytes F0 9D 84 9E, str2 will be … uhm … maybe 6 bytes ED A0 B4  ED B4 9E. Somehow this super-simple ABNF *(%x0-10FFFF) has unleashed weird system-dependent behaviour. The weirdness doesn’t come from the ABNF; it’s from somewhere else.

JSON is not as weird as *(%x0-10FFFF). You can interpret JSON specs to imply an unpaired surrogate can only appear as an escape sequence. So that’s a higher-layer concern that doesn’t impact your UTF-8/16/32 encode/decode part at all. ECMA 404 doesn’t mention lone surrogates at all. And even if you go for max 16-bit backward compatibility it is far more easily described as “accept ill-formed UTF-16 if you can; send ill-formed UTF-16 with lone surrogates escaped if you must”.

That’s why I really dislike *(%x0-10FFFF) being used as a shorthand for how JSON can support Unicode plus arbitrary 16-bit data (ie ill-formed UTF-16). JSON specs muddle through by defining JSON text not the logical strings a text can represent; by having an escaping mechanism; by being a bit loose; by leaving semantic decisions to the specific processor; and with paragraphs on interop.

If draft-bray-unichars defines *(%x0-10FFFF) as a standalone item labelled a character repertoire or subset of Unicode characters it need oodles more text to explain the weirdness above. Treating surrogates on par with control code and noncharacter “problematic” code points does not come close.



>>>Perhaps this document should reference <https://unicode.org/reports/tr17/#Strings> (note authors), which covers similar territory.

>>Thanks for noticing.

>>The need for transient states that are discoverable (that is, not fully encapsulated) is a big reason why many specs are not tighter.

>>However, there are points in a protocol where strings aren't in a transient processing state, and here the full restrictions should apply (and be specified).
> Yes. The problem here is that JSON can transmit stuff resembling these: "For example, strings in Java, C#, or ECMAScript are Unicode 16-bit strings, but are not necessarily well-formed UTF-16 sequences." I also mentioned it because it says "A string data type is simply a sequence of code units.", which matches ECMA-404 pretty well.
>
> Here, the distinction between "string" and UTF-8/UTF-16/UTF-32 is clearly drawn. To use James' example:
>
> ---
>
> It does not make sense for a spec to define:
>  unicode-code-point = %x0-10FFFF
>  string = *unicode-code-point
>
> ---
>
> It seems to me that TR17 defines "string" this way. Which is not to say that I recommend sending these things over the internet, just that it can happen. I think the draft does a decent job discouraging this one, but I guess it will have to be yet clearer.

No. TR17 says a Unicode 16-bit string is *(%x0-FFFF). Quite different from *(%x0-10FFFF).
TR17 does look good.

--
James Manger



General