Re: [I18ndir] draft-bray-unichars-01

[Asmus Freytag <[e-mail]>]

On 8/29/2023 7:50 PM, Tim Bray wrote:
>
>
> On Aug 29, 2023 at 5:22:03 PM, Asmus Freytag <[e-mail]> wrote:
>> On 8/29/2023 5:05 PM, Tim Bray wrote:
>>> Asmus, these are great! Thanks. If this document goes forward I 
>>> think it will include almost all of what you suggest.
>> I would be thrilled. I think it's useful to move this forward (with 
>> changes).
>>>
>>> My favorite input is (7) - Neither co-author is in love with 
>>> “Basic”, a better suggestion would be greeted with joy.
>>
>> "Minimally restrictive" describes what you are aiming for. You are 
>> restricting everything that has no business being interchanged and no 
>> more.'
>>
> Standing alone, that makes sense. But compared to the other options in 
> the draft, such as those used by JSON & XML, we are 
> /more/ restrictive. So there also could be a case for “maximally 
> restrictive”. Maybe just “Restricted Code Points”?  Hmm, not obvious.

Just intended to get your thinking started. The full descriptive name 
would be:

"Assigned plus Assignable code points minus 'useless' control codes".

Alternatively:

"Complete present and future Graphic, Format and Private-Use characters, 
together with three essential control codes".

None of those trip off the tongue, but again, it's useful to stare at 
them a bit. Perhaps that might triggers something.


Now, the collection that you describe is the one that is useful for text 
representation for interchange. You exclude the control code which are 
(except for the three "essential" ones) really about protocol support 
and not the text itself, and you exclude all the stuff that shouldn't be 
interchanged (noncharacters, surrogates).

"Default Unicode Subset for interchanging Text".  (DUSFIT).

You see where I'm going, using Default instead of Basic, echoing similar 
subsets for "Default Identifiers". Both are geared to a specific purpose 
and both carry in the name that they might be further customized (e.g. a 
non-default text subset might exclude Private use, to ensure that all 
characters have a shared semantic).

It might be possible to change the initials a bit to make them 
pronounceable.

What do you think?

A./

>> Something like this then sets up the idea that there are other 
>> scenarios where the likely direction is "more restrictive", except 
>> perhaps some protocols that do assign meaning to certain controls and 
>> therefore rely on them -- we all think that these should be the 
>> exception; in other words, if you picked something that indicates an 
>> endpoint (like "minimally") the nice feature would be that most other 
>> specifications would tend to subset.
>>
>> For example, by restricting private use.
>>
>> A./
>>
>> PS: you might think about
>>
>> (8)
>>
>> Suggest a standard methodology for specifying subsets other than the 
>> "XML" or "minimally restrictive" ones / or at least discuss some 
>> concepts around this area, or point to specs in and out of ITETF that 
>> create limited subsets for various purposes.
>>
>> There's always a tension of defining data formats that don't impose 
>> practical restrictions on the text content and situation where 
>> general text is not what you need (but pure ASCII strings don't cut 
>> it either).
>>
>>
>>>
>>> On Aug 29, 2023 at 12:10:42 PM, Asmus Freytag <[e-mail]> 
>>> wrote:
>>>> Comments on the draft (in no particular order or priority).
>>>>
>>>> (1)
>>>> "Since the C0 controls include zero and the 32 smallest integers, 
>>>> they are likely to occur in data as a result of programming errors."
>>>>
>>>> In the way the Unicode standard refers to characters "the control 
>>>> codes" are the characters, so "the C0 control codes include" would 
>>>> be read that way. What you are referring to here are the integer 
>>>> values, corresponding to a code point. Suggest rewording because 
>>>> "zero" as a digit is not a control code and while it's possible to 
>>>> figure out what must be meant, it's a needless ambiguity.
>>>>
>>>> How about "Since the code points for C0 controls include the 32 
>>>> smallest integers including zero..."
>>>>
>>>> (2)
>>>> I recommend thinking not of "problematic code points" but of 
>>>> "problematic code point types" as per this definition from the 
>>>> Unicode Glossary. /
>>>> /
>>>>
>>>>     /Code Point Type
>>>>     <https://www.unicode.org/glossary/#code_point_type>/. Any of
>>>>     the seven fundamental classes of code points in the standard:
>>>>     Graphic, Format, Control, Private-Use, Surrogate, Noncharacter,
>>>>     Reserved. (See definition D10a in Section 3.4, Characters and
>>>>     Encoding <https://www.unicode.org/versions/latest/ch03.pdf#G2212>.)
>>>>
>>>> This would let you also address the private-use problematic in 
>>>> interchange (even if you conclude they might be useful enough not 
>>>> to restrict them by default, they suffer from the same lack of 
>>>> consensus interpretation as the controls).
>>>>
>>>> (3)
>>>>
>>>> "reserved for internal use" should be "reserved for internal use by 
>>>> application" (as opposed to internal use by the standard).
>>>>
>>>>
>>>> (4)
>>>> This subset has the advantage of excluding surrogates, which can 
>>>> never add any value and have the potential to cause problems.
>>>>
>>>> should be reworded a bit to:
>>>>
>>>> "This subset has the advantage of excluding surrogates, which are 
>>>> not assigned to any characters, and thus can never add any value.
>>>> They have the potential to cause problems, for example it is not 
>>>> possible to represent them individually in UTF-8."
>>>>
>>>> Rationale for this suggestion is to be slightly more specific, so 
>>>> the reader comes away with the conclusion that the "can never add 
>>>> any value" is based on well-founded reasons and not editorial 
>>>> opinion by the writers.
>>>>
>>>> (5)
>>>> The ABNF in section 4.1: the comments are confusing.
>>>>
>>>> (6)
>>>> I would like to refer you to table 2-3 in the Unicode Standard. The 
>>>> "Basic Unicode Characters" that you propagate consist of what 
>>>> Unicode considers "Assigned" code points plus "Reserved", with the 
>>>> modification that you are subtracting the "useless controls".
>>>>
>>>> Reserved code points are sometimes referred to as "assignable code 
>>>> points" (for example on the bottom of page 30 in Unicode 15.0.0). 
>>>> That makes that subset the combination of "assigned" plus 
>>>> "assignable" code points. (Which then, modulo "useless" controls, 
>>>> corresponds to the bulk of the basic set.)
>>>>
>>>> I would suggest to explicitly relate your definition to those terms.
>>>>
>>>> (6)
>>>> The definition "useful controls" is currently buried in the text 
>>>> and there's not even a header to locate the definition. Because 
>>>> this (or the complementary definition of "useless" ones) is the 
>>>> only value added piece over "assigned + assignable", I suggest 
>>>> elevating the definition.
>>>>
>>>> I'm not going to quibble about the very opinionated naming of the 
>>>> concept.
>>>>
>>>> (7)
>>>> The term "Basic" is an interesting choice. Because the set is 
>>>> anything but "basic" -- it includes all code points that can be 
>>>> maximally assigned, except for 61 C0 / C1 Controls. The only part 
>>>> of the set that is "basic" is in fact the subset of control codes.
>>>>
>>>> What you have defined is the "maximally useful set of Unicode code 
>>>> points for data interchange, absent a protocol defining specific 
>>>> control code semantics".
>>>>
>>>>
>>>> Hope you find some of these comments useful,
>>>> A./
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 8/29/2023 9:37 AM, Tim Bray wrote:
>>>>> Hello I18ndir (anyone still here?), Paul Hoffman and I just 
>>>>> submitted draft-bray-unichars-01 - our AD Francesca Palombini 
>>>>> suggested we notify this list: 
>>>>> https://datatracker.ietf.org/doc/draft-bray-unichars/
>>>>>
>>>>> This draft fell out of a conversation originally provoked by this 
>>>>> errata report: https://www.rfc-editor.org/errata/eid7603 
>>>>> <https://www.rfc-editor.org/errata/eid7603>
>>>>>
>>>>> It revealed a distressing lack of consensus about Unicode 
>>>>> characters and code points and character repertoires. I feel 
>>>>> personally bad because I am the editor of a couple of RFCs that 
>>>>> are open to criticism on this front.
>>>>>
>>>>> So, this tries to say “here’s how an RFC should specify which 
>>>>> Unicode characters it supports”.  We think this would be useful to 
>>>>> multiple groups almost immediately, including especially those who 
>>>>> don’t realize the area can be problematic.
>>>>>
>>>>> Anyhow, the purpose of this note is to ask your advice on how to 
>>>>> take this forward. We think this works well as an individual 
>>>>> submission.   We don’t *think* it needs a working group, but 
>>>>> that’s not our call. The draft doesn’t express any opinions about 
>>>>> best practices, it just points out several alternative character 
>>>>> repertoires, provides ABNF, and discusses their trade-offs.  None 
>>>>> of them are wrong.
>>>>>
>>>>
>>