Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

[Asmus Freytag <[e-mail]>]

On 9/9/2023 4:50 PM, Tim Bray wrote:
> On Sep 9, 2023 at 1:56:47 PM, Rob Sayre <[e-mail]> wrote:
>> 5. Refining Character Repertoires
>>
>> The IETF typically uses well-known data formats such as JSON, I-JSON, 
>> CBOR, YAML, and XML. These formats have default character 
>> repertoires. For example, JSON allows member names and string values 
>> to include any Unicode code points, including all the problematic 
>> types; the following is a legal JSON document:
>
> I’m OK with Rob’s redraft, if only to lose the fuzzy word 
> “typically”.  Anyone object?

Unless there are obvious exceptions that otherwise would have to be 
covered, I hate using weasel words myself.

Sometimes, "almost always" is a better fit than "typically" - again, 
depending on whether it fits the fact.

> .
>> {"example": "\u0000\U0089\uDEAD\u7FFFF"}
>>
>> The value of the "example" field contains the C0 Control NUL, the C1 
>> Control "CHARACTER TABULATION WITH JUSTIFICATION", an unpaired 
>> surrogate, and the noncharacter U+7FFFF. It is unlikely to be useful 
>> as the value of a text field. It cannot be serialized into legal 
>> UTF-8, but many libraries will silently parse this and generate an 
>> ill-formed UTF-8 string. Implementors must be prepared to deal with 
>> these sorts of problematic code points
>>
>> [ The first part, "unlikely to be useful as the value of a text 
>> field", is good. But, the next part mixes "legal" and "ill-formed", 
>> and I don't think that is a good idea. There is still a lowercase 
>> requirement after that, and I think I disagree. Implementors do not 
>> have to be "prepared to deal with these sorts of problematic code 
>> points". Maybe: "Some messages will contain these 
>> problematic code points". That is true, but you don't have to deal 
>> with them. ]
>
> Yeah, I’m pretty sure you do, but by “deal with” I include rejecting 
> such input docs, e.g. by raising a well-known exception with a helpful 
> error message.  I guess that should be spelled out explicitly?

I agree that there's a difference between somehow "doing the right 
thing" when input is broken and various ways of "rejecting" ill-formed 
input. I wrote a lengthy comment about this in a message that didn't 
have you on the CC, so I don't know whether it got to you (I later 
forwarded some).

Strong recommendation about how to process ill-formed input is 
beneficial from a security perspective. Look up what Unicode writes 
about doing with ill-formed UTF-8 as an example of what to do when you 
can't throw an exception.


>
>> It is unlikely that anyone specifying a new data format would choose 
>> to allow this character repertoire.
>>
>> [ Instead: The JSON character repertoire is too permissive, so it's 
>> best for new specifications to require that the contents of member 
>> names and string values contain only Useful Assignables (see Section 
>> 4.2). ]
>
> I prefer the current language. Anyone prefer Rob’s or to propose 
> another option?

I think that the proposed language has the advantage of being more 
concrete. That makes it easier to follow and to remember. (Unless, of 
course, the additional detail is incorrect).

I think "anyone specifying a new data format built on JSON should/SHOULD 
limit the contents...." might be a way to grab the best of both.

>
>> Then, I got to the end, and noticed that "character repertoire" might 
>> not be the best choice. "Character encoding" or "character set"? 
>> "Vocabulary"? No shade for the authors here, writing about language 
>> itself is really difficult.
>
> I personally want to stay with “repertoire” if only by extension from 
> its well-understood meaning when applied to encoding systems.
>>
>>
>