Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

[Asmus Freytag <[e-mail]>]

On 9/8/2023 1:16 PM, Rob Sayre wrote:
> On Fri, Sep 8, 2023 at 12:38 PM Tim Bray <[e-mail]> wrote:
>
>     See https://www.ietf.org/archive/id/draft-bray-unichars-03.html
>
>     A bunch of minor corrections and improvements, thanks to everyone
>     for that, especially James Manger for noticing that the ABNF was
>     entirely wrong in one place.
>
>     The word “useless” has been replaced by “legacy”.
>
I like.

On re-reading, I note that I keep stumbling over this statement

"While the inclusion of unassigned code points in text data is 
undesirable, it is difficult to specify that it should be avoided, 
because unassigned code points regularly become assigned as new 
characters are added to Unicode."

Small, closed character sets are a luxury of the past. They are 
unrealistic, because they assume a static universe. A comprehensive 
character set like the Unicode Standard must be able to grow. Not only 
because it intends to be universal, with many writing systems not yet 
cataloged, but also because existing writing systems change. Western 
Europe alone, despite its well established use of the Latin script, has 
seen two new characters, the € and the capital sharp s. When you have a 
closed set, your only way to react to such changes is to switch 
character sets.

A minimal fix to the text might be to change "is undesirable" to "may 
have its drawbacks".

Now, there are cases where it is necessary to restrict the use of 
Unicode characters to the subset that represents a known version. This 
is usually the case if data has to be normalized (and must stay 
normalized, something that cannot be guaranteed for unassigned code 
points). For an example, see IDNA2008.

Which leads to a suggestion for a fifth set (really a series), which 
would be "versioned-assigned". It would include all code points assigned 
as of a given version, minus the "legacy controls". That is a nice 
repertoire to use whenever you need to normalize your data and guarantee 
that their normalization does not change. (As it turns out, you can 
migrate your repertoire, over time, to later versions, as long as data 
that is normalized is free of unassigned code points relative to the 
version of the normalization tables.)


>
>     I think the feedback was pretty clear that the draft needed to be
>     more opinionated; just because we document the existence of the
>     default JSON repertoire (“all the code points”) doesn’t mean that
>     anyone should use it in the present or future. So, introduced a
>     new section “Refining Character Repertoires” to highlight those
>     issues and offer a suggestion.
>
>
> This one is tougher and correct. Fully in favor.
>
> I would change
>
> "These numbers are used to represent the characters in computer memory 
> and storage systems and, in specifications, to specify the allowed 
> repertoires of Unicode characters."
>
> to
>
> "These numbers are used to represent the allowed repertoires of 
> Unicode characters."

This change makes it sound like that is their only use. For that reason, 
I like the original formulation better.

I can see where it could be a little confusing in the other direction, 
because the original formulation might imply that internal 
representation is directly in terms of those numbers, whereas it usually 
is in terms of a transformation format, whether UTF-8 or UTF-32. Now, we 
don't want to delve deeply into a discussion of transformation formats, 
code units and the like.

"These numbers underlie the representation of characters in computer 
memory, storage systems and data transmission. In specifications they 
are used..."

To me, that better reflects the central nature of this concept, but 
removes the implication that they are used "as is" in data representation.

>
> Other commenters have said the "useful" term is not that great. I 
> agree, but I can't think of anything better. In particular, I thought 
> "no, people really do use NULL". Maybe "text-characters"? IDK, up to 
> the editors.

There are two types of data formats. Those that can contain NULL and 
those that cannot.

NULL, where used, always has a special meaning. The days where it could 
be inserted into a datastream with "null effect" are long over, event 
thought that was among the original use cases.

Most people don't use VT or FF (or NEL) but there are legacy data 
formats that do.

Perhaps, the best thing would be to acknowledge that there may be a need 
to be more permissive, if and when lossless conversion from legacy data 
is needed. A useful discussion would contain a recommendation to what 
spec authors should do.

The section on "Refining Character Repertoires" currently represents 1/2 
of the coin -- it is limited to a discussion on how to be more 
restrictive. It could benefit from discussing the other half: what about 
citing something like "useful assignables" *plus* selected controls, 
instead of either allowing all controls, or using one of the wider 
repertoires that drags in all the non-characters.

I think that would be a useful extension of the work.


>
> thanks,
> Rob
>
>