Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-04.txt

[Asmus Freytag <[e-mail]>]

On 9/22/2023 11:44 AM, Rob Sayre wrote:
> On Wed, Sep 20, 2023 at 9:56 AM Rob Sayre <[e-mail]> wrote:
>
>     Well, it depends on whether you need to interoperate with the Web.
>     This can mean not only detecting the "toxic waste" code points,
>     but accepting them. That seems straightforward to me. I think the
>     other protocols and subsets are a better idea in other situations.
>
>
> ...
>
> Since strings in some languages and systems are Unicode strings, but 
> not necessarily well-formed UTF-8 or UTF-16, it can make sense to 
> transmit text of this sort.
>
> thanks,
> Rob
>
I think we need to distinguish three different needs.

(1) identifiers (names)

(2) clean definition of text content fields

(3) string data type (which is also used internally and may have 
transient states)

These use cases need different repertoires, and have different constraints.

Identifiers benefit from a limited repertoire, often sharply limited, to 
aid in reliable recognition. Being mnemonic devices, there's no need to 
be able to represent all edge cases for all languages, as there would be 
for text. There are many other specifications that provide identifier 
repertoires, such as UAX#31, IDAN2008, domain-specific repertoires, and 
so on. For the purpose here, it would be useful to mention that there is 
a class of use cases that does need stricter repertoire limits than 
discussed here and that what is discussed here isn't recommended for 
those purposes.

A clean text content field will be specified as text data in a 
well-formed encoding form and repertoire that excludes internal use as 
well as "useless" controls. As an output specification, that would be 
complete, as an input specification, it raises the issue of what to do 
with input that fails to fully conform. (Being ill-formed, or exceeding 
the repertoire). We have useful discussion on that.

Then there's a the string data type. We don't need to define it, because 
Unicode defines it already, but it would be useful to put it into 
perspective. As defined, the string data type includes code points that 
can only ever be used in internal processing, and it includes ill-formed 
encoding forms, because they will normally occur in transient states 
during processing and may need to be temporarily held when receiving 
input (before and during verification). Those are the reasons that 
generic string data types are not restricted, unlike text content fields 
conforming to a protocol. (There may be protocols that should be defined 
on the basis of strings and not text content and for those, the 
recommendations here would not apply. And that should be explained).


Being explicit about these use cases and their different needs, even if 
the suggested repertoires only address the middle case, prevents readers 
from bringing their assumptions based on different use cases.

A./