Re: [I18ndir] Fwd: New Version Notification for draft-bray-unichars-04.txt

[Asmus Freytag <[e-mail]>]

On 9/17/2023 4:06 PM, Tim Bray wrote:
> On Sep 15, 2023 at 2:01:48 PM, Asmus Freytag <[e-mail]> wrote:
>>
>> This time I looked only at the diffs and sometimes a bit of adjacent 
>> text.
>>
>> This first one is not major, but a small fix would avoid a 
>> contradiction in terms.
>>
>>> The numbers assigned to Unicode characters are called “code points”;
>> This is backwards, as can be seen by "unassigned" code points. Easy 
>> to fix:
>
> You are correct that there’s a problem here.  However, as we 
> previously established (on the art@ list only, so you probably didn’t 
> see it), there’s not really a “backward" and “forward". The Unicode 
> Standard is inconsistent on “assignment”, saying both that integer 
> code points are assigned to characters, and that characters are 
> assigned to code points. See 
> https://mailarchive.ietf.org/arch/msg/art/3s70K1uHFguqeL8DFkIjiNfP5ZY/

OK - would have been helpful if I had seen that earlier.

And I'm not surprised to see some variability in usage in the Unicode 
Standard.

However, the fact that the underlying mapping is bidirectional makes it 
possible to speak of "assigned code points" while simultaneously 
considering the code point a piece of, as you call it, meta data on the 
character.

In this case "the numbers used to represent Unicode character are ...." 
would also work, as you found out re: "representation".


>
> The current draft talks about code points being assigned to characters 
> and is consistent, end to end. That’s because we felt that the 
> abstract character was the important thing, and the code point is a 
> useful piece of attached metadata.  I’m strongly of the opinion that 
> we should be consistent even if the Unicode standard isn’t.  So we can 
> either address this problem by reversing the direction of assignment 
> end-to-end and adopting your suggestion, or by modifying the language 
> to make it clearer what code points are - and in fact the definition 
> from the Unicode standard (which we quote later) says explicitly it's 
> the numbers in the code space 0-0x10FFFF.  I’ll figure out which gives 
> a better result.

The nub of my comment was that you had assignment one way, but talk 
about "un/assigned" code points which then is weird.

You may be able to retain the sense you have by talking a bit about the 
assignment and also introducing representation.

It might not hurt giving the definition of encoded character.

I think you have all the information in hand to come to a good resolution.

A./


>> New text on dealing with problematic code points:
>>
> …
>>> In applying the recommendations of RFC19413 for text fields 
>>> containing ill-formed UTF-8, for example, the recommendations must 
>>> be applied to the field as a whole, not on the character or byte 
>>> level. In fact, silently ignoring an ill-formed part of a string is 
>>> a known security risk. Responding to that risk, [UNICODE] section 
>>> 3.2 ....
>
> Looks good to me.
>>
>> The last paragraph is overselling RFC9413, because the phrasing 
>> conceivably implies that it contains guidance specific to code 
>> points, when it is more generically concerned with problematic input. 
>> It also doesn't flow particularly well.
>>
>> You could move it at the head of Section 5, with tweak.
>>
>>> Problematic code points are an example of problematic input. 
>>> [RFC9413], "Maintaining Robust Protocols", provides a thorough 
>>> discussion of error-handling options when choosing a strategy for 
>>> dealing with problematic input. Different types of problematic code 
>>> points cause different issues.
>>>
>>> Noncharacters....
>>
>> (I'm also suggesting adding a sentence to make the transition)
>>
> Will do something like what you suggest.
>>