Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

[Asmus Freytag <[e-mail]>]

On 9/10/2023 12:05 AM, Rob Sayre wrote:
>
>
>>>     And then in 8.
>>>
>>>      8.  String and Character Issues
>>>      8.1.  Character Encoding
>>>         JSON text exchanged between systems that are not part of a
>>>         closed ecosystem MUST be encoded using UTF-8 [RFC3629].
>>
>>     As Rob Sayre said above, the proposed document probably has to
>>     address the issue of JSON escapes and emphasize that they are not
>>     relevant to code point subsets.
>     No they are. If you are in a JSON-based environment, but have
>     restricted your repertoire, then even if JSON allows an escape,
>     it's invalid if it violates your restriction. And your new
>     specification SHOULD require something definite and drastic to
>     happen in that case.\
>
>
> That is exactly the point. The current draft mentions "Transformation 
> Formats", but doesn't mention that these transformation formats can 
> and do further encode questionable Unicode via escape sequences. 
> The draft should mention it.
>
> Unfortunate, but true. So, you can have a perfect UTF-8 document that 
> represents a bunch of unpaired surrogate code points.

The term "transformation format" is not formally defined  in the Unicode 
Standard. It is noted in chapter 3 as being a (somewhat ambiguous) alias 
for two other terms, encoding format and encoding scheme. ("schemes" 
have a fixed byte order for code units and "forms" are generic mappings 
to an integral type irrespective of byte order).

The key is that the definition of both schemes and forms specifies them 
as a mapping from a single code point (actually: scalar value) to a 
"sequence of code units", while the term "transformation format"  as 
defined in the Unicode glossary is a more general mapping of character 
sequence to sequence of code units.

    /Transformation Format
    <https://www.unicode.org/glossary/#transformation_format>/. A
    mapping from a coded character sequence to a unique sequence of code
    units (typically bytes).

The draft currently states:

> Unicode describes a variety of "transformation formats", ways to 
> encode code points in bytes of computer memory. A survey of 
> transformation formats is beyond the scope of this document.

If we accept the term "transformation format" as an alias for the terms 
that are actually defined in the standard (encoding form or scheme) then 
that usage of "transformation format" _excludes _escape sequences and 
similar mappings where the source of the mapping can be more than one 
code point. The UTF-8, UTF-16 and UTF-32 encoding forms will be formally 
called out as "standard encoding forms" in the next revision of the 
Unicode Standard.

There are non-standard ones, such as UTF-7, CESU-8 and whatnot, but none 
of them have escape sequences either.

While the in memory form of a string syntax with an escape sequence is 
technically an example of a "transformation format" as that term is 
described in the glossary, such a syntax is neither an encoding form, 
nor an encoding scheme, and those are the things the Unicode Standard 
actually describes. Because, according to the definition, the escape 
sequence would represent multiple code points and is therefore not a 
direct representation of a character under one of the standard encoding 
forms.

Rob's comments make clear that using the term "transformation formats" 
in the way quoted above can lead to misunderstandings.

I sympathize with the desire not to bring in the entire formalism, so 
the following suggestion might suffice to limit the sense of 
"transformation format" that is intended here.

> Unicode describes a variety of "transformation formats", ways to 
> *uniquely *encode *each **scalar value* into bytes of computer memory. 
> A survey of transformation formats is beyond the scope of this document.

If you like to avoid the term "scalar value" - it is synonymous to 
"non-surrogate code point".

A./


>
> thanks,
> Rob
>
>