Re: [I18ndir] [art] Just uploaded draft-bray-unichars-03

["Manger, James" <[e-mail]>]

> Also, I bet that if I had a JSON text {“example”: “\uDEAD”} and fed it to JSON-to-CBOR converters, a lot of them would emit CBOR containing ill-formed UTF-8.

CBORMapper from the Jackson library doesn’t. (I don’t have much experience with CBOR converters, but Jackson is a very popular Java JSON lib).

com.fasterxml.jackson.core.JsonGenerationException: Invalid surrogate pair, starts with invalid high surrogate (0xDEAD), not in valid range [0xD800, 0xDBFF]
                at app//com.fasterxml.jackson.core.JsonGenerator._reportError(JsonGenerator.java:2849)
                at app//com.fasterxml.jackson.dataformat.cbor.CBORGenerator._invalidSurrogateStart(CBORGenerator.java:1723)
                at app//com.fasterxml.jackson.dataformat.cbor.CBORGenerator._encode2(CBORGenerator.java:1703)
                at app//com.fasterxml.jackson.dataformat.cbor.CBORGenerator._encode(CBORGenerator.java:1660)
                at app//com.fasterxml.jackson.dataformat.cbor.CBORGenerator._writeString(CBORGenerator.java:1451)
                at app//com.fasterxml.jackson.dataformat.cbor.CBORGenerator.writeString(CBORGenerator.java:877)
                at app//com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)
                at app//com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:479)
                at app//com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:318)
                at app//com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4719)
                at app//com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3987)

> BTW this assertion that “UTF-8 can’t include surrogates”, which has been made repeatedly, needs to be taken with a grain of salt. The UTF-8 procedures for converting between code points and byte sequence work perfectly well for surrogates and a whole lot of software out there will silently convert both ways. The UTF-8 in question is in fact not well-formed nor does it conform to the definition of UTF-8, but it exists in the wild and it can’t really be defined as “non-existent”.

A whole lot of software does NOT silently accept ill-formed UTF-8|16|32.
So even if some software does, and even though ill-formed UTF-8 exists, unicode-code-points is still totally unsuitable as a “character repertoire”.

 The default repertoire of JSON is not unicode-code-points since JSON excludes controls except tab, newline and carriage return. Given this spec distinguishes useful-assignables from unicode-scalar-values it should distinguish JSON’s actual subset from unicode-code-points if it is going to mention JSON.

> You’re thinking of XML? https://datatracker.ietf.org/doc/html/rfc8259#section-7 says that C0 controls must be expressed in \u notation but they’re allowed.

JSON can describe a string that has a U+0000 char (using an escape sequence of 6 ASCII chars). But the JSON itself is not allowed to include a U+0000. So the character repertoire for the JSON protocol (the ABNF for the JSON data format) does not allow U+0000. json.org has ’0020’ . ‘10FFFF’; ECMA 404 says “any code point except “ or \ or control character”; RFC8259 has %x20-21 / %x23-5B / %x5D-10FFFF. None of these are unicode-code-points.

 I agree that “many libraries will silently parse” "\uDEAD", but I’m not sure how many “generate an ill-formed UTF-8 string”. In Java, for instance, "\uDEAD".getBytes("UTF-8") returns a single byte 0x3F “?” – it’s valid UTF-8, just no longer a 1-to-1 representation of the in-memory unpaired surrogate.

> Wow. I had no idea. As with many aspects of Java+Unicode, this feels deeply wrong. It should either round-trip or throw a damn exception. Anyhow, that ship sailed a long time ago.  I think we should include the Java example to illustrate another way that surrogates can lead to breakage.

Java correctly detects an attempt to UTF-8-encode a lone surrogate as wrong. CharsetEncoder.encode does throw a MalformedInputException. But Java helpfully offers 3 ways to handle malformed-input and unmappable-character errors: CodingErrorAction.IGNORE|REPLACE|REPORT. The REPLACE option drops the erroneous input, appending a replacement value – which is what String.getBytes(Charset) is defined to do.

You can (with at least some libraries) round-trip JSON with lone surrogates in Java – but the output has to be JSON (with the lone surrogate again represented with an escape).


General