Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-04.txt

[Asmus Freytag <[e-mail]>]

On 9/18/2023 1:31 PM, Rob Sayre wrote:
>
>
> On Mon, Sep 18, 2023 at 1:17 PM Asmus Freytag <[e-mail]> 
> wrote:
>
>     On 9/18/2023 11:09 AM, Rob Sayre wrote:
>>     On Mon, Sep 18, 2023 at 10:51 AM Rob Sayre <[e-mail]> wrote:
>>
>>         On Mon, Sep 18, 2023 at 7:05 AM Manger, James
>>         <James.H.Manger=[e-mail]> wrote:
>>
>>             For understandable reasons, JSON supports both *(%x0-D7FF
>>             / %xE000-10FFFF) and *(%x0-FFFF) (arbitrary 16-bit data)
>>             as models for the logical strings it can represent.
>>
>>
>>         ECMA-404 is clear: "JSON syntax describes a sequence of
>>         Unicode code points." and the discrepancy between this text
>>         and RFC8259 is what motivated this document. The document
>>         also seems to fairly clearly recommend against using this
>>         production if you can help it.
>>
>>
>>     Perhaps this document should reference
>>     <https://unicode.org/reports/tr17/#Strings> (note authors), which
>>     covers similar territory.
>>
>     Thanks for noticing.
>
>     The need for transient states that are discoverable (that is, not
>     fully encapsulated) is a big reason why many specs are not tighter.
>
>     However, there are points in a protocol where strings aren't in a
>     transient processing state, and here the full restrictions should
>     apply (and be specified).
>
> Yes. The problem here is that JSON can transmit stuff resembling 
> these: "For example, strings in Java, C#, or ECMAScript are Unicode 
> 16-bit strings, but are not necessarily well-formed UTF-16 sequences." 
> I also mentioned it because it says "A string data type is simply a 
> sequence of code units.", which matches ECMA-404 pretty well.
>
> Here, the distinction between "string" and UTF-8/UTF-16/UTF-32 is 
> clearly drawn. To use James' example:
>
> ---
>
> It does not make sense for a spec to define:
>   unicode-code-point = %x0-10FFFF
>   string = *unicode-code-point
>
> ---
>
> It seems to me that TR17 defines "string" this way. Which is not to 
> say that I recommend sending these things over the internet, just that 
> it can happen. I think the draft does a decent job discouraging this 
> one, but I guess it will have to be yet clearer.
>
The disconnect may be that Unicode needs to define "string" in ways that 
are not limited to interchange.

A string data type may transiently be in state that is not well-formed, 
or may be able to hold ill-formed input data before verifying its status.

Unicode, in other words, has to be universal covering ALL possible uses 
of a "string". The same is not true for a protocol that defines a record 
with data fields. Those fields don't need to be able to accommodate 
ill-formed data.

The exception might be a protocol that is "just a pipe". In that case, 
the responsibility for enforcing well-formedness belongs to (usually) 
the receiver connected to that "pipe". Such a protocol may well want to 
allow "all unicode code points".

A./