Re: [I18ndir] Fwd: Re: Working Group Last Call: Structured Headers for HTTP

["Asmus Freytag (c)" <[e-mail]>]

On 2/5/2020 5:23 PM, John C Klensin wrote:
> Asmus,
>
> In at least my case and I assume in Patrik's and John Levine's,
> when I said "ASCII", I might better have said "the ASCII graphic
> subset of Unicode", or "the Basic Latin repertoire of Unicode",
> or "Unicode code points in the range U+0030 (or perhaps U+0020)
> through U+007E (or maybe +007A), i.e., a repertoire, not a
> specific coding standard.

Right; so far I'm with you.

I do agree that as far as repertoires go, it has some simplicity to it, 
although even with "ASCII" you run into questions of how you want to 
handle case.

>    Given that all of the documents to
> this particular discussion are pushing UTF-8 and that a Unicode
> string from that repertoire in UTF-8 is indistinguishable at the
> octet level from a ASCII string encoded by right-justifying each
> seven-bit ASCII code point in an eight bit byte (octet) with a
> leading zero bit, the shortcut should be obvious.  But you are
> correct in that we were talking about repertoire, not encoding
> or choice of character set.

I also agree that the ASCII repertoire has the nice property that it 
maps to the same octets whether UTF-8, Latin-1 of many other character sets.

Whether a protocol needs to support a CCS declaration is a separate 
question. Beyond explicitly identifying data as utf-8, that is (rather 
than implicitly via choice of protocol).

Allowing the protocol to be expressed in any other CCS than utf-8 is 
something that in today's environment should require to clear a 
substantial hurdle (by promising a really clear win in compatibility 
with key installed technology and which is not outweighed by imposing 
costs donwstream).

>
> That said, it may be worth remembering that, independent of what
> operating systems may or may not do, early Web specifications
> were written assuming ISO 8859-1 and that there are almost
> certainly some applications out there that assume that CCS when
> they see octets with the leading bit on.  The ASCII repertoire
> as described above is still a proper subset, but, because 8859-1
> (and 8859-x more generally) are not UTF-8-compatible, the
> ability to define the CCS and encoding in use may still be
> necessary even if the necessity is waning.

Precisely the reason to make sure new protocols don't add to this 
problem other than with a gun to their heads, so to speak. Definitely 
not on the "there might be" level of reasoning, I'd think.

A./

>
> best,
>    john
>
>
>
> --On Wednesday, February 5, 2020 14:50 -0800 Asmus Freytag
> <[e-mail]> wrote:
>
>> (didn't go out to the list when I first sent it)
>>
>> When I read the word "exception" I always think of this:
>>
>> When we built the first Unicode-enabled OS (Windows NT), we
>> had a long discussion of which "strings" in the OS needed to
>> be Unicode.
>>
>> Some thought that there was a clear dividing line between data
>> and what would be called "protocol values" in another context.
>>
>> Some of the latter did look like they were easily limited to
>> ASCII; but everywhere we found "exceptions". There might be a
>> set of enumerable tokens, but it allowed extended values that
>> were network or file identifiers.
>>
>> After exhaustively researching everything, the conclusion was
>> that every single string in the OS had to be Unicode (and
>> making any exceptions was either not possible, or not worth
>> the effort).
>>
>> However, while all strings were encoded in Unicode, not all
>> string values were allowed. While file names could be
>> localized (within the limits of file system syntax), some of
>> the enumerated strings were left limited to the ASCII set in
>> repertoire (even if encoded in Unicode).
>>
>> Reading this discussion (and I'm sorry I don't have the time
>> right now to properly delve into the details) it seems that a
>> natural recommendation would be to require Unicode for any
>> native representation and, if necessary (or possible), limit
>> the repertoire.
>>
>> This also requires a definition of the matching protocol for
>> all strings that are to be matched as part of the protocol (or
>> should be searchable). For any format, that would cover issues
>> of casing, white space handling etc., but for Unicode, by
>> necessity, that also requires defining the normalization form
>> to be used.
>>
>> A./
>>
>> PS: given how few systems these days natively operate in any
>> character set other than Unicode, I am always astonished at
>> the length to which people go to justify not making something
>> native Unicode. They just pick up conversion issues when they
>> use platform libraries to do any work or display.
>>
>