Re: [Json] In "praise" of UTF-16

[Carsten Bormann <[e-mail]>]

Hi Anders,

> On Aug 31, 2019, at 17:39, Anders Rundgren <[e-mail]> wrote:
> 
> I-D dealing with canonical JSON serialization is currently in the IETF ISE queue

I don’t really care much about this draft, as I have pointed out.
(Mainly not because deterministic encoding is wrong, but because the reasons you want to define a “standard” for it are wrong.  But that has been said already in different messages.)

So please do go ahead and taint your scheme further with UTF-16 legacy stuff.

For those people who missed the early history of ISO 10646 (aka Unicode):
Unicode was supposed to be the 16-bit character set to end all 8-bit character sets.
We were all supposed to move to UCS-2 (a 2-byte, 16-bit representation where the standards bodies couldn’t even decide the endianness, so we actually have two of those).

One important language designed in the early-to-mid-1990s got stuck with this: Java.  As did Windows.
Pioneers get shot full of arrows.  
The .NET system designed for Windows as a competitor to the JVM (Java Virtual Machine), as well as JavaScript (which is not Java but usurped the name), also inherited 16-bit characters.

Then it dawned people that 16 bits were not enough, and after a short excursion to a 31-bit system, UTF-16 was invented as a hack to accept more than 63488 [sic] codepoints, with the built-in problem that the “astral” codepoints (beyond 2**16) were infrequent enough that people built endless amounts of bugs into software that used it.  

The 16-bit legacy embedded into Java and its contemporaries could not be stamped out since.  The main reason is that while UTF-16 is a hack that hurts, it does not hurt enough to sustain deployment of a fix (remember that even Python is still struggling this very year with getting its Unicode fixes deployed).

JSON of course inherited the 16-bit legacy from JavaScript in its string escape scheme, but otherwise did come out pretty much unharmed and managed to focus on UTF-8, like everything else relevant to interchange.

Defining a deterministic encoding scheme (“canonicalization”) for JSON in 2019 that needs a detour through UTF-16-land looks like a cruel joke.  If you only ever care about Java and its contemporaries, it may actually seem practical to you.  Given that most map keys will be ASCII anyway, people will certainly find shortcuts (also known as sleeping interoperability problems) around the issue.  A protocol designer would not touch this with a 16-foot pole.

Grüße, Carsten