[art] Re: draft-bray-unichars-09 review

[Rob Sayre <[e-mail]>]

On Tue, Oct 8, 2024 at 9:52 AM Tim Bray <[e-mail]> wrote:

>
> ### Where is UTF-16 used?
>>
>> ```
>> 178   A surrogate which occurs in text encoded in any transformation
>> format
>> 179   other than UTF-16 has no meaning and may cause malfunction in
>> 180   software that encounters it.  In particular, it is impossible to
>> 181   represent a surrogate in well-formed UTF-8.
>> ```
>>
>> Some guidance here on where to expect UTF-16 could help explain where to
>> expect surrogates.
>>
>
> UTF-16 is essentially never used on the wire. I for one have never seen it
> in my decades of work in this space.  Does anyone have a counter-example?
> Or even better, does anyone know of an RFC that says “you MUST NOT put
> UTF-16 on the wire.”? Because thankfully, nobody does.
>

Its main use today is to send around XML files that are in UTF-16.

https://datatracker.ietf.org/doc/rfc2781/referencedby/

If you run "grep -r" on /Applications on macOS, you'll find a lot of those.
It's because the OS strings were always UTF-16 or UCS-2 until recently, but
it's a gradual unwind. I think the industry realized that O(1) access to
strings, as if they were an array of numbers, was not really necessary. So,
then UTF-8 is better.

It will never go away, though. See this header that is probably shipped
with your operating system of choice:

https://opensource.apple.com/source/WTF/WTF-7606.3.4.0.2/icu/unicode/platform.h.auto.html

"C++11 and C11 require support for UTF-16 literals" ...so now it's baked in.

thanks,
Rob