Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

[Rob Sayre <[e-mail]>]

I also went and looked at the Fetch standard.* This is like XMLHttpRequest
but newer, and it was specified after I got out of the browser game. The
easiest thing to read is the Servo Rust version of this (all browsers are
amazing, but are usually written in pretty gnarly C++).

https://github.com/servo/servo/blob/bd9c17234c330e6a2f01fdc472a9dd81a0ea05c5/components/script/body.rs#L822

If you dig through the WHATWG document, you'll find that the error mode is
"replacement". So, you can see there that the browser gets a buffer,
creates UTF-8 that may contain replacement characters (the escape sequences
are not dealt with yet), converts that to UTF-16, and then calls
JSON.parse, where many of our not-so-favorite code points might occur via
escape sequences.

I think this is a good example, and validates the approach in the draft.
Here, we see that not only do web browsers send questionable strings, they
also process them that way:

1) lossy UTF-8 decoding
2) convert to UTF-16
3) convert that to possibly questionable Unicode after processing escape
sequences

Gross, but you can still get sensible Unicode through there by following
this draft. It is also more efficient to use correct UTF-8, because the
decoder won't allocate as much. If you avoid escape sequences, the JSON
parser will also go faster. I don't write this as if browsers are the last
word, but there sure are a lot of them, and it's tough to change every
website, so they tend to converge on convenient solutions. They can still
accommodate change (like HTTPS or QUIC), but it's a tall order.

thanks,
Rob

* https://fetch.spec.whatwg.org/#ref-for-dom-body-json

On Mon, Oct 2, 2023 at 5:32 PM Rob Sayre <[e-mail]> wrote:

> On Mon, Oct 2, 2023 at 5:21 PM Manger, James <James.H.Manger=
> [e-mail]> wrote:
>
>> draft-bray-unichars
>> <https://datatracker.ietf.org/doc/html/draft-bray-unichars> §3 “Dealing
>> with problematic code points” suggests “replacing problematic code points
>> with "�" (U+FFFD, REPLACEMENT CHARACTER)” (or signalling an error, but I’ll
>> only talk about the replacement option in this email).
>>
>
> It's probably not worth dwelling on. But I will get into this one, at the
> risk of sounding like a major dork. This issue is already decided, so we
> are only describing what already happens:
>
>
> https://doc.rust-lang.org/std/string/struct.String.html#method.from_utf8_lossy
>
> The reason the unichars document should not unreservedly recommend a
> replacement character is because that's lossy.
>
> thanks,
> Rob
>
>