Re: [Json] Generators

[Marcos Caceres <[e-mail]>]

Hi Stefan,  

On Monday, June 10, 2013 at 11:59 AM, Stefan Drees wrote:

> On 2013-06-10 12:10 CEST, Marcos Caceres wrote:
> > On Monday, 10 June 2013 at 10:49, Markus Lanthaler wrote:
> > > On Monday, June 10, 2013 9:38 AM, Stefan Drees wrote:
> > > > 5. Generators
> > > >  
> > > > A JSON generator produces JSON text that MUST strictly conform to
> > > > the JSON grammar.
> > >  
> >  
> >  
> >  
> > Please try to avoid "wish list" style conformance requirements. If you
> > want "strictly" conforming documents generated (whatever that may mean),
> > then please write the algorithm to produce such documents. For example:
> >  
> > "When generating JSON, as JSON generator MUST run the steps to generate JSON text.
> >  
> > The steps to generate JSON text are given by the following algorithm:
> > 1. …
> > 2. …
> > "
>  
>  
>  
> if I only once had the power to write my wish lists that strong and  
> concise, I would be a very happy man I guess :-)
>  
> We will not have a conformance section and we had until now no step by  
> step description but only an "integral" check, where the result had to  
> conform to the ABNF grammar of JSON.
> So the following message is the best we can state:
> If you are a wannabe X generator, ensure your output strictly conforms  
> to our X grammar.
>  
> Retro-inventing of stepwise solutions/recipes simply not having been  
> around for all these years and then acting as if they had, to me  
> personally gives a strong smell and is far off the charter, isn't it?

I haven't read the charter, tbh … I literally stumbled onto this list yesterday :)  However, I'm worried that overly simplistic or optimistic conformance requirements will just force people to go and look at ES6 to actually write the parsing and generation algorithms (or worst, they will just find some other random implementation out there and copy that). That could greatly diminish the value of this specification as it will leave implementers scratching their heads about how to implement either a parser or a generator.  

Also, given the legacy of JSON, it should be possible to see which parser behavior "won" (e.g., Doug's original JSON.js?) - and model this spec off that.  

Additionally, I suspect few people actually implement an ABNF parser - whatever is expressed in ABNF will have to be translated into code for a specific implementation. You can essentially shortcut that by specifying the algorithm and making implementer's lives easier.  That's not to say that the ABNF should not be there (it helps the reader understand components of the JSON grammar) - but the reality of the situation should be acknowledged (ABNF is an editorial aid, but rarely, AFAIK, a tool used in implementations in it's pure form to do syntactical checking).   
   
> > > > Generators MUST NOT duplicate names in objects if they can avoid or
> > > > detect such duplication.
> > > >  
> > > > """
> > > [...]
> > > > This "MUST NOT duplicate ... if" reads pretty soft for implementers,
> > > > but should be the only consensual non-breaking way out of the
> > > > historical dilemma of not enforcing it for generators in the first place.
> > >  
> >  
> >  
> >  
> > I agree. The above "if they can avoid [it]" if pretty lousy spec text.
> > Please define the algorithm instead to produce the results you want.
>  
>  
>  
> I might be tempted to change the sequence to a more natural looking one  
> "detect before avoid" but then, if an implementation is not able to  
> avoid it, why should it bother to detect or not detect? So the strongest  
> excemption is noted first: A policy or a use case hinders me in ensuring  
> the "uniqueness wish" for names in objects.
>  
> On a personal note: If we avoid wordings like "pretty lousy spec text"  
> in situations where many people are trying to retrofit best possible  
> constraints into an area where these completely missed for years, to  
> avoid further future "leakage" and without breaking reality that would  
> be surely helpful, wouldn't it?

Yes, I'm sorry; I didn't mean any offense. I just meant that it was a conditional MUST, and that's not how MUST should be used (as it means one parser can do one thing and another can do another thing … which is an interoperability anti-pattern). This would, obviously, be bad if I'm sending one JSON representation from the client and it gets transformed into a different one on the server and the spec says that this behavior is ok.   

On the other hand, if the spec has to say, "well, that's the reality right now… and here are some guidance on how to avoid that problem without breaking clients and servers:" that might have to do.  

> So IMO the terrain for the pair Generators/Parsers has three main  
> boundary conditions:
>  
> a) the solutions for the generator / parser pair must match w.r.t.
> chaining the MUST/SHOULD/MAYs of each.
>  
> b) the generator in reality always transforms some storage pattern
> (JSON RFC is agnostic of that!) into a JSON text conforming to
> our JSON ABNF. Thus, the Generator has not to deal with invalid
> "input": piece of cake :-)
>  
> c) the parser in reality always transforms some JSON text into some
> storage pattern (JSON RFC is agnostic of that!). The parser has to
> deal with invalid input claiming to be JSON text!
>  
> in this thread we discuss b) and try to not mess with a).
>  
> Until now it has already been stated on this list, that in reality (and  
> rightly so):
>  
> 1. generator A can't detect duplicate names in objects
>  
> 2. generator B uses duplicate names to annotate values with comments
>  
> 3. generator C uses duplicate names to somehow stream, send historical
> data or deltas.
>  
> Given that A, B and C all are valid JSON implementations of Generators  
> we will not retro-invalidate them.

Agree. Whatever is specified needs to be backwards compatible on the parser side. Perhaps it's futile to assume a "generator", because most of the time it's people who are writing JSON by hand (so it doesn't make any sense to use RFC2119 language relating to any "generator" because people can't be a conformance class). Best you can do with generators is to assume some "best practice" acknowledging that JSON is used in the scenarios A, B, C, and possibly others to help dictate the behavior of the parser.  

> In that light, the above MUST NOT ... if (describe the known exceptions)  
> approximates best at least IMO the many variants proposed.

Thanks for the context. It's indeed a tough one :)  
> > For excellent examples of how to write a specification using
> > step-based conformance, please see the ES6 spec. See the parsing
> > algorithm for JSON here - you basically want a language neutral version
> > of that, right?:
> >  
> > http://people.mozilla.org/~jorendorff/es6-draft.html#sec-15.12.2 ...
>  
> Thank you for this nice description of the steps that a specific  
> languages JSON Parser will have to perform to conform to that language  
> specification.
>  
> No, I think we do not want a language neutral version of the JSON.Parse  
> description in ES6.
>  
> Yes, we do want ES6s JSON expectations to remain valid if they were  
> before, but we do not have a mandate to re-invent a step by step JSON  
> Generator (or Parser) recipe, as that would need to be checked against  
> all valid (as of now) implementation recipes of all existing parsers and  
> would blow up this nice little successfull super star RFC quite a bit or  
> two ;-)

Ok, looks like I need to find the charter. However, concerns about the size of the spec should not ever enter the discussion - if it needs to be a long, bloated spec, to achieve interoperability and clarity - then that is the "right thing to do" (tm)… or, put another way, please make the spec only as long as it needs to be to achieve interoperability and the goals, and no longer :)