Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

[Asmus Freytag <[e-mail]>]

On 9/9/2023 4:44 PM, Tim Bray wrote:
> On Sep 9, 2023 at 9:58:43 AM, Steffen Nurpmeso <[e-mail]> wrote:
>>
>> In 2.2 i would not give the count on code point types.
>> Instead i would only give the problem statement "among Unicode
>> code point types .. are questionable".  This seems more generic.
>
> Left to myself I’d leave it as-is, but I don’t care that much, if 
> anyone else agrees with Steffen I don’t mind changing it.
>
I tend to disagree. Unicode explicitly defines 7 types and by giving 
that count you reinforce that these are predefined and not some generic 
typology.
>
>> In 2.2.2.2 i would not say "legacy controls", and that they are
>> "mostly obsolete".  ECMA-48 is very alive in at least the POSIX
>> aka Linux world, for many purposes, for example terminal
>> interaction.  "Likely to occur in data as a result of
>> a programming error"?  Any preformatted Unix manual page will come
>> with lots of CSI sequences, or backspace-based ones.
>> ASCII NUL is the base of ISO C-style strings.  In fact many
>> network protocols (not enough!!) still seem to use
>> KEY=VALUE\0KEY=VALUE\0\0 style transports.
>
> So, in section 23.1 of [UNICODE] it says "There are 65 code points set 
> aside in the Unicode Standard for compatibility with the C0 and C1 
> control codes defined in the ISO/IEC 2022 framework.”  Some shreds of 
> ISO 2022 practices may survive in legacy systems but I am pretty sure 
> they are not relevant to any protocol or data format the IETF might 
> work on. This clearly feels like “legacy” and “mostly obsolete” to me. 
> (For those who don’t know, ISO 2022 worked with the "ISO-Latin" 8859-1 
> code pages to shift between them in the middle of a string. It is now 
> forgotten for a good reason.) I’d never heard of ECMA 48 and I am 
> dubious that that use of control codes would be appropriate in 
> anything the IETF would specify these days.

Yes, for text data, these are legacy.

However, there are modern implementations of terminals and their 
protocols would use ECMA48. (There's an ISO standard that matches ECMA48 
and which is cited in the Unicode Standard, I think it's 6942).

I keep suggesting that the way out for this is to acknowledge that some 
data streams have needs for these code points and if your spec must 
include them, but you would like to avoid surrogates and non-characters, 
that the answer would be to construct your own subset by extending the 
Useful-Assignables with as many control codes as you need (in an 
explicit list).

That gets your draft out of the business of creating subsets for edge 
cases, while simultaneously staying relevant even for people writing 
such specs. (I'd see that as a win-win).



>
>> In 5.:
>>
>>  [JSON..] It cannot be serialized into legal UTF-8, but many
>>  libraries will silently parse this and generate an ill-formed
>>  UTF-8 string. Implementors must be prepared to deal with these
>>  sorts of problematic code points.
>>
>> But RFC 3629 is very clear and says in 3. (being lengthy)
>
> You are correct, but the assertion in 5. is also empirically correct, 
> this is exactly what many (most?) existing libraries will do. 
> Presumably on the basis of Postel’s law?
>
>> So even the weird JSON "string" can be made valid UTF-8, one just
>> has to walk around the corner.  (Possibly.)
>
> In fact, does that happen? I haven’t seen it but I suppose it is 
> somewhat possible.
>
>> Sorry, but _I_ do not get that JSON supports _that_ "string",
>> RFC 8259, 7.:
>>
>>   To escape an extended character that is not in the Basic Multilingual
>>   Plane, the character is represented as a 12-character sequence,
>>   encoding the UTF-16 surrogate pair.
>
> Once again, you are probably correct, but the caution that 
> implementers will have to be “prepared to deal with” this is true, 
> because of what implementations do.  You are correct that it would 
> probably be better if JSON parsers were stricter.

The focus is subsets.

Data formats for which implementations allow (whether sanctioned or not) 
to specify strings that are outside the repertoire of the subset (even 
outside some subset like the useful assignables) basically allow the 
creation of ill-formed data.

And that should be made very explicit.

In the context of a syntax that allows an escape to represent an 
noncharacter, any data stream that contains that escape would be 
ill-formed with respect to the "useful-assignable" subset repertoire. 
And would need to be treated as such.


>
>> And then in 8.
>>
>>  8.  String and Character Issues
>>  8.1.  Character Encoding
>>     JSON text exchanged between systems that are not part of a
>>     closed ecosystem MUST be encoded using UTF-8 [RFC3629].
>
> As Rob Sayre said above, the proposed document probably has to address 
> the issue of JSON escapes and emphasize that they are not relevant to 
> code point subsets.
No they are. If you are in a JSON-based environment, but have restricted 
your repertoire, then even if JSON allows an escape, it's invalid if it 
violates your restriction. And your new specification SHOULD require 
something definite and drastic to happen in that case.
>
>
A./