Re: [I18ndir] draft-bray-unichars-01

[Tim Bray <[e-mail]>]

On Aug 29, 2023 at 5:22:03 PM, Asmus Freytag <[e-mail]> wrote:

> On 8/29/2023 5:05 PM, Tim Bray wrote:
>
> Asmus, these are great! Thanks. If this document goes forward I think it
> will include almost all of what you suggest.
>
> I would be thrilled. I think it's useful to move this forward (with
> changes).
>
>
> My favorite input is (7) - Neither co-author is in love with “Basic”, a
> better suggestion would be greeted with joy.
>
> "Minimally restrictive" describes what you are aiming for. You are
> restricting everything that has no business being interchanged and no more.'
>
Standing alone, that makes sense. But compared to the other options in the
draft, such as those used by JSON & XML, we are *more* restrictive.  So
there also could be a case for “maximally restrictive”.  Maybe just
“Restricted Code Points”?  Hmm, not obvious.

> Something like this then sets up the idea that there are other scenarios
> where the likely direction is "more restrictive", except perhaps some
> protocols that do assign meaning to certain controls and therefore rely on
> them -- we all think that these should be the exception; in other words, if
> you picked something that indicates an endpoint (like "minimally") the nice
> feature would be that most other specifications would tend to subset.
>
> For example, by restricting private use.
>
> A./
>
> PS: you might think about
>
> (8)
>
> Suggest a standard methodology for specifying subsets other than the "XML"
> or "minimally restrictive" ones / or at least discuss some concepts around
> this area, or point to specs in and out of ITETF that create limited
> subsets for various purposes.
>
> There's always a tension of defining data formats that don't impose
> practical restrictions on the text content and situation where general text
> is not what you need (but pure ASCII strings don't cut it either).
>
>
>
> On Aug 29, 2023 at 12:10:42 PM, Asmus Freytag <[e-mail]>
> wrote:
>
>> Comments on the draft (in no particular order or priority).
>>
>> (1)
>> "Since the C0 controls include zero and the 32 smallest integers, they
>> are likely to occur in data as a result of programming errors."
>>
>> In the way the Unicode standard refers to characters "the control codes"
>> are the characters, so "the C0 control codes include" would be read that
>> way. What you are referring to here are the integer values, corresponding
>> to a code point. Suggest rewording because "zero" as a digit is not a
>> control code and while it's possible to figure out what must be meant, it's
>> a needless ambiguity.
>>
>> How about "Since the code points for C0 controls include the 32 smallest
>> integers including zero..."
>>
>> (2)
>> I recommend thinking not of "problematic code points" but of "problematic
>> code point types" as per this definition from the Unicode Glossary.
>>
>> *Code Point Type <https://www.unicode.org/glossary/#code_point_type>*.
>> Any of the seven fundamental classes of code points in the standard:
>> Graphic, Format, Control, Private-Use, Surrogate, Noncharacter, Reserved.
>> (See definition D10a in Section 3.4, Characters and Encoding
>> <https://www.unicode.org/versions/latest/ch03.pdf#G2212>.)
>>
>> This would let you also address the private-use problematic in
>> interchange (even if you conclude they might be useful enough not to
>> restrict them by default, they suffer from the same lack of consensus
>> interpretation as the controls).
>>
>> (3)
>>
>> "reserved for internal use" should be "reserved for internal use by
>> application" (as opposed to internal use by the standard).
>>
>>
>> (4)
>> This subset has the advantage of excluding surrogates, which can never
>> add any value and have the potential to cause problems.
>>
>> should be reworded a bit to:
>>
>> "This subset has the advantage of excluding surrogates, which are not
>> assigned to any characters, and thus can never add any value.
>> They have the potential to cause problems, for example it is not possible
>> to represent them individually in UTF-8."
>>
>> Rationale for this suggestion is to be slightly more specific, so the
>> reader comes away with the conclusion that the "can never add any value" is
>> based on well-founded reasons and not editorial opinion by the writers.
>>
>> (5)
>> The ABNF in section 4.1: the comments are confusing.
>>
>> (6)
>> I would like to refer you to table 2-3 in the Unicode Standard. The
>> "Basic Unicode Characters" that you propagate consist of what Unicode
>> considers "Assigned" code points plus "Reserved", with the modification
>> that you are subtracting the "useless controls".
>>
>> Reserved code points are sometimes referred to as "assignable code
>> points" (for example on the bottom of page 30 in Unicode 15.0.0). That
>> makes that subset the combination of "assigned" plus "assignable" code
>> points. (Which then, modulo "useless" controls, corresponds to the bulk of
>> the basic set.)
>>
>> I would suggest to explicitly relate your definition to those terms.
>>
>> (6)
>> The definition "useful controls" is currently buried in the text and
>> there's not even a header to locate the definition. Because this (or the
>> complementary definition of "useless" ones) is the only value added piece
>> over "assigned + assignable", I suggest elevating the definition.
>>
>> I'm not going to quibble about the very opinionated naming of the
>> concept.
>>
>> (7)
>> The term "Basic" is an interesting choice. Because the set is anything
>> but "basic" -- it includes all code points that can be maximally assigned,
>> except for 61 C0 / C1 Controls. The only part of the set that is "basic" is
>> in fact the subset of control codes.
>>
>> What you have defined is the "maximally useful set of Unicode code points
>> for data interchange, absent a protocol defining specific control code
>> semantics".
>>
>>
>> Hope you find some of these comments useful,
>> A./
>>
>>
>>
>>
>>
>>
>>
>>
>> On 8/29/2023 9:37 AM, Tim Bray wrote:
>>
>> Hello I18ndir (anyone still here?), Paul Hoffman and I just submitted
>> draft-bray-unichars-01 - our AD Francesca Palombini suggested we notify
>> this list: https://datatracker.ietf.org/doc/draft-bray-unichars/
>>
>> This draft fell out of a conversation originally provoked by this errata
>> report: https://www.rfc-editor.org/errata/eid7603
>>
>> It revealed a distressing lack of consensus about Unicode characters and
>> code points and character repertoires. I feel personally bad because I am
>> the editor of a couple of RFCs that are open to criticism on this front.
>>
>> So, this tries to say “here’s how an RFC should specify which Unicode
>> characters it supports”.  We think this would be useful to multiple groups
>> almost immediately, including especially those who don’t realize the area
>> can be problematic.
>>
>> Anyhow, the purpose of this note is to ask your advice on how to take
>> this forward. We think this works well as an individual submission.   We
>> don’t *think* it needs a working group, but that’s not our call. The draft
>> doesn’t express any opinions about best practices, it just points out
>> several alternative character repertoires, provides ABNF, and discusses
>> their trade-offs.  None of them are wrong.
>>
>>
>>
>