Re: [Json] Scope: Wire format or runtime format?

[Norbert Lindenberg <[e-mail]>]

Hi Martin,

On Jun 16, 2013, at 21:17 , Martin J. Dürst wrote:

> Hello Norbert,
> 
> On 2013/06/14 7:47, Norbert Lindenberg wrote:
>> In looking over older messages on this list, I found a message that made clear to me why we're having this endless discussion about Unicode surrogates - it's because we're not clear whether we're designing a wire format or a format that also for use at runtime:
>> http://www.ietf.org/mail-archive/web/json/current/msg00355.html
> 
> This is kind of the first time I have heard the term "format for use at runtime" in this context. Of course there are formats used at runtime (internal representations of number and strings, for example), but I think the term "runtime format" only confuses our discussion. In my understanding, JavaScript does not have or use JSON as a runtime format. If that has changed, then please tell us.

Maybe you have a better term for it: What I mean in the ECMAScript context is the format that the functions JSON.parse and JSON.stringify parse and produce [1]. The format is based on the JSON grammar, represented in ECMAScript string values, that is, sequences of UTF-16 code units. ECMAScript doesn't specify wire formats (it doesn't even have any sort of I/O system), and has no control over what applications do with the strings before passing them to parse() or after obtaining them from stringify(). In many cases applications will get them from or send them to some I/O system (such as XMLHttpRequest in the browser or the file system module in Node.js), but they might modify them in between, or do something completely different.

>> Some people are coming from the runtime point of view, especially ECMAScript, where it's accepted practice to use ill-formed UTF-16 or even non-text in strings. At least the ill-formed UTF-16 is legitimized by section 2.7 of the Unicode standard.
> 
> "Accepted practice" is probably going a bit too far, and giving the wrong impression. My understanding is that the Unicode standard accepts this for efficiency reasons, not because it's in anyway inherently useful. For ECMAScript, we have to add history to efficiency, but still I hope it's considered bad practice to actually use ill-formed UTF-16.

"Grudgingly accepted practice"? Unfortunately, considering it bad practice to use ill-formed UTF-16 or to represent JPEGs as string values doesn't make it go away.

>> Other people are coming from the wire protocol point of view, where clean formats are expected, in particular well-formed Unicode code unit sequences according to section 3.9 of the Unicode standard.
>> 
>> So which one shall it be?
>> 
>> If we adopt the wire protocol point of view
> 
> To me, it's very clear that we are describing (we are not designing; that has happened a long time ago, and very implicitly) a wire format, or something close to a wire format (somebody mentioned JSON embedded in an (e.g.) EUC-JP HTML page (*)).
> 

>> and require well-formed code unit sequences,
> 
> For practical reasons, I think we shouldn't go there. But we should make it very clear (in the base spec, not relegated to some "best practices" document) that using lone surrogates is a bad idea, and that senders and receivers MAY reject such data (e.g. because of security concerns).

John Cowan pointed to RFC 3629, which prohibits the use of surrogate code points in UTF-8 [2]. Do you think that even for a wire format we can ignore that prohibition?

>> then ECMAScript will have to define its own extension of JSON
> 
> I very much hope we can avoid that. I very much hope that ECMAScript can tolerate that lone surrogates are often if not always a bad idea, even if they may sometimes happen for historical and efficiency reasons.

The ECMAScript specification generally doesn't discuss good or bad ideas; it just specifies the behavior of implementations. The specified behavior is that unpaired surrogates in strings are passed through by both parse() and stringify(), and parse() unescapes escape sequences for all surrogate values, paired or not.

Personally, I agree that unpaired surrogates are in most cases a bad idea.

[1] http://ecma-international.org/ecma-262/5.1/#sec-15.12
[2] http://www.ietf.org/mail-archive/web/json/current/msg00831.html

Regards,
Norbert