Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

[Rob Sayre <[e-mail]>]

On Sun, Sep 10, 2023 at 7:26 PM Manger, James <James.H.Manger=
[e-mail]> wrote

>
>
> Java correctly detects an attempt to UTF-8-encode a lone surrogate as
> wrong. CharsetEncoder.encode does throw a MalformedInputException. But Java
> helpfully offers 3 ways to handle malformed-input and unmappable-character
> errors: CodingErrorAction.IGNORE|REPLACE|REPORT. The REPLACE option drops
> the erroneous input, appending a replacement value – which is what
> String.getBytes(Charset) is defined to do.
>
>
>
> You can (with at least some libraries) round-trip JSON with lone
> surrogates in Java – but the output has to be JSON (with the lone surrogate
> again represented with an escape).
>

It is so much worse than that. see:
https://stackoverflow.com/a/7921064/5965814

I don't think we should document any of this, because there is an economic
incentive to not do anything of the sort. The reason is that you can avoid
allocations on the server if everyone is keeping it clean in correct UTF-8.
I think newer JVM/Java things get this right, too.

I know this piece of annoying trivia, because we measured UTF-8 to
UTF-16/UCS2 processing costs in Mozilla Firefox. These were
surprisingly high. At the time, the OS strings meant that we were going to
pay either way, and it was not worth refactoring Firefox strings to use
UTF-8 internally.

These days, the OS strings are getting better.

thanks,
Rob