Re: [Json] Proposed minimal change for strings

["Manger, James H" <[e-mail]>]

> * Paul Hoffman wrote:
> >Proposal 2 (prohibit unescaped surrogates):
> >
> >In section 1 (Introduction):
> >  A string is a sequence of zero or more Unicode scalar values
> [UNICODE].
> 
> Section 1 describes the JSON data model and the proposed change would
> prohibit escaped unpaired surrogates, implying that something like
> 
>   JSON.stringify([ JSON.parse('["..."]')[0].substring(0, 140) ]);
> 
> occasionally returns a string that is not JSON text, even if the input
> to `JSON.parse` is proper JSON text, when there is a non-BMP character
> in just the wrong place (`substring` counts UCS-2 code units).
> 
> With JSON.stringify this can be avoided using a replace function like
> 
>   JSON.stringify(obj, function(key, value) {
>     if (typeof value == 'string') {
>       return value.replace(/.../g, "\uFFFD");
>     }
>     return value;
>   })
> 
> but then you have dataloss, unnecessarily so because you might just be
> putting an object into a datastore for later use and ecmascript doesn't
> mind unpaired surrogates, and the code is slower and harder to read.


It is only "unnecessary dataloss" when you are certain a value will only be handled by ECMAScript -- in which case relying on ECMAScript's (reasonable) extension beyond JSON is ok.
On the other hand, what happens if the datastore understands JSON and tries to store the non-dataloss values as UTF-8? That is likely to fail. In this case "dataloss" that the app controls (and can avoid by not splitting a non-BMP character) is much better that unpredictable behaviour (works or fails) from different JSON-compliant datastores.


> It seems pretty clear to me that decoders are a better place to deal
> with unpaired surrogate code points. In environments where they pose no
> problem you probably want them to roundtrip, in others substituting is
> a better option, and in yet others you might want to fail hard on them.
> 
> Prohibiting unpaired surrogate escapes would place the burden to avoid
> them on senders, and I have not seen in argument why that is a better
> option, all things considered.

I think your previous paragraph provides the argument. If different decoders can legitimately roundtrip, substitute, or fail on parsing an unpaired surrogate escape than senders need to be warned (in big flashing lights) of this non-interoperable behaviour. The best way to do that is to exclude unpaired surrogate escapes from the specification of JSON.

--
James Manger