Re: [Jsonpath] Some Comments ...

[Stefan Gössner <[e-mail]>]

I copied these comments forward to Github ... for better reading and 
further commenting.

https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath/issues/54

I would also like to try out that cool Github discussion tab, Daniel 
wrote about.

thanks
--
sg


Am 26.02.2021 um 19:30 schrieb Stefan Gössner:
> Hello List,
>
> It has been important to go through this list threads carefully. In 
> fact I should have done that at first. Now I can understand the 
> current draft and appreciate the work already done much better.
>
> I collected some citations (important from my point of view) with 
> comments already in Markdown.
>
>
> ## Title of the specification
>
> > JSONPath: A query language for JSON data.
> *(Carsten Bormann)*
>
> > I think I’d slightly prefer the term “syntax” to “language” because 
> “query language” has a smell of various things that end with the 
> letters “Q” and “L”.  But not passionate about that.
> *(Tim Bray)*
>
> > JSONPath: A query syntax for JSON.
> > Another wild-card idea: JSONPath: Query expressions for jSON
> *(Tim Bray)*
>
> > The beauty of this is that the plural form “query expressions” 
> implies a set of expressions, so it implies “language”.  It’s indeed 
> more than the grammar/syntax of those, so why not talk about the 
> expressions as a whole.  This also makes it possible to just use “for 
> JSON”, without going into detail what these query expressions operate on.
> *(Carsten Bormann)*
>
> There seems to be an agreement for "*JSONPath: Query expressions for 
> JSON*". I like that also.
>
> ## Terminology
>
> > My own view is that the terminology should stay consistent with RFC
> 8259, and that the word "object" should not be used for items that are
> not JSON objects in the sense of RFC 8259.
> *(Daniel P)*
>
> > To Carsten's point about what we call things, the number of 
> distinguished
> terms per RFC8259 is pretty small: JSON text, value, object, array, 
> number,
> string.  Having spent quite a bit of time specifying JSON DSLs, I find 
> that
> using just those terms doesn't seem to get in the way or cause 
> problems, so
> I'd argue that we should stick to them (and build up to higher-level
> constructs as required for JSONPath).
> >
> > … oh, and I forgot the very useful "member".
> *(Tim Bray)*
>
> > … and “element” (the things in arrays). *(Carsten Bormann)*
>
> > The problem with JSON value is that it also can be quite confusing 
> due to the usual use of that term.  Pointing to a tree and saying “the 
> values inside that tree” is not going to be felt as equivalent to “the 
> set of all subtrees of that tree, including the tree itself”.  But if 
> JSON value is the only term we have, it has to be.  Hence my 
> preference to talk about data items when I mean the items themselves 
> and not their “value”.
> *(Carsten Bormann)*
>
> > I think the key difficulty is whether each (key, value) pair in an 
> object is "a thing" that can be identified and manipulated and 
> potentially returned. (If we're talking analogies, then it's analogous 
> to an attribute node in the XDM model).
> *(Michael Kay)*
>
> > ECMA-404 uses "name/value pair", which is what I understand the term
> "member" to mean (Douglas Crockford uses "member").
> *(Daniel P)*
>
> > I think the term “union” is poor. If we think of it as concatenation 
> of results, then the result is as expected.
> *(Glyn Normington)*
>
> I understand, that within RFC8259 we have JSON values of different 
> types. They are structured somehow, which is not so much of interest 
> here.
>
> But while querying that structure with JSONPath it is vitally 
> important to identify that hierarchical structure as a tree. So in 
> fact we build up a higher-level construct here. We also need to call 
> "the things" in the tree somehow. I was able to identify
>
> * "node" or "item" of a tree
> * "member" of an object
> * "name/value" or "key/value" pair alias "member"
> * "element" of an array
>
> but could not see an agreement here.
>
> I agree to Glyn calling the term "union" poor (s. below).
>
>
> ## Differentiation from JSON Pointer (JSONPath draft charter)
>
> > I anticipate being asked "Why is JSON Pointer not sufficient?" 
> Indeed its abstract says:
> >
> >   JSON Pointer defines a string syntax for identifying a specific 
> value within a JavaScript Object Notation (JSON) document.
> >
> >... which sounds awfully similar.  If we could include a sentence about
> that, or a link to an answer, that might be helpful.
> *(Murray S. Kucherawy)*
>
> > No - it's not similar in concept, they're separate things. If you 
> really wanted to mention JSON Pointer, you could say something like 
> "Note that while JSON Pointer (RFC xxxx) is already standardised, it 
> is designed to provide a reference to a single, specific part of a 
> JSON document, whereas JSONPath provides the ability to query a 
> document and potentially return multiple values."
> *(Mark Nottingham)*
>
> >The short answer is that JSON pointer is good if you already know the 
> structure of the JSON data item you want to point into, and you want 
> to point to exactly one position in there.  If you need to do 
> something that is closer to a “search” (which might also result in 
> multiple positions), JSONPath gives you more rope.
> *(Carsten Bormann)*
>
> +1
>
> ## References to XPath
>
> > I wonder if the analogies between XPath and JSONPath are going to be 
> helpful, or whether they're actually dangerous by implying 
> equivalences between constructs that are in fact somewhat different?
> *(Michael Kay)*
>
> > I tend to agree. Although JSONPath was inspired by XPath, I wouldn't
> want to confuse the JSONPath spec by going into detailed comparisons at
> the risk of contradicting the normative text.
> *(Glyn Normington)*
>
> > Someone on StackOverflow today asked a question about JSONPath; they 
> called it (and tagged it) XPath, we really don't want that kind of 
> confusion.
> >
> > In addition, the reference to the XPath specification in 6.2 is out 
> of date, and the comparison with XPath in Table 2 is very approximate 
> and the terminology inaccurate: for example there is a mention of 
> "node sets", which exist in XPath 1.0 but not in XPath 2.0, yet the 
> citation is to XPath 2.0. For someone who knows the semantics of XPath 
> the comparison raises all sorts of questions about sorting of results 
> into document order, elimination of duplicates etc, which are 
> complications this spec can well do without. (Though some answers are 
> needed, for example if ..store..price matches the same price in more 
> than one way, do you get more than one result? And if not, what does 
> "the same price" actually mean?)
> *(Michael Kay)*
>
> It seemed to be important in 2007, while argumenting to have something 
> like XPath for JSON. If nowadays the terminology used has changed 
> significantly with XPath 2.0 and 3.0, we better leave that comparison 
> table 2 out. I am quite passionless here.
>
> ## Array Slice Operator
>
> > Thanks! The ABNF for an array slice in that reference
> > ```
> > integer = [%x2D] (%x30 / (%x31-39 *%x30-39))
> >
> > array-slice = [ integer ] ws %x3A ws [ integer ]
> >                     [ ws %x3A ws [ integer ] ]
> >                              ; start:end or start:end:step
> > ```
> > is consistent with JMESPath, Python, and my understanding of
> ECMASCRIPT 4.
> > *(Daniel P)*
>
> > Did anyone else have an opinion on the behaviour of slices such as 
> [::0]?
> The current draft allows this and says it returns an empty array, but 
> there
> is good reason to say it should error so that the slice operation is then
> consistent with Python slicing. See below for more context.
> *(Glyn Normington)*
>
> It's good having read this thread and thus understand the current 
> draft much better. I like the decision to be consistent with Python 
> and also getting an empty selection set with `step=0`.
>
> FYI: there is a recent proposal for adding slice notation syntax to 
> JavaScript, currently at stage 1 of the TC39 process.
>
> https://github.com/tc39/proposal-slice-notation
>
> Interestingly it won't have a step argument ...
>
> https://github.com/tc39/proposal-slice-notation#why-doesnt-this-include-a-step-argument-like-python-does 
>
>
> ... because of syntax collision with the new `this-binding` syntax 
> proposal `::`
>
> https://github.com/tc39/proposal-bind-operator
>
> However, we should not let us influence by this.
>
> ## Unions
>
> > I don't think any implementation would remove duplicates from a path
> such as `"$.store.book"`. I believe this is only somewhat controversial
> in the context of unions [,]. The name "union" suggests that distinct
> values be returned, compare with SQL unions. But Stefan Goessner's
> implementation doesn't do that, it concatenates all results that meet
> each criteria. There are a few JSONPath implementations that produce
> real unions with no duplicates instead of concatenated results, but I
> don't think that's the consensus.
> *(Daniel P)*
>
> > I think the term “union” is poor. If we think of it as concatenation 
> of results, then the result is as expected.
> *(Glyn Normington)*
>
> > I agree with that comment, but it's partly because I'm used to SQL 
> UNION,
> which is different. I prefer the JMESPath term for an analogous 
> construct,
> MultiSelect List, 
> https://jmespath.org/specification.html#multiselect-list.
> *(Daniel P)*
>
> Introducing the union operator `[,]` simply was meant an analogon to 
> XPath's operator `'|'`. I cannot tell, if it was a simple combination 
> of node sets in Xpath 1.0 or a true union without duplicates. I 
> obviously was not aware of that subtle (essential ?) union 
> characteristic.
>
> So I fully agree to Glyn Normington's '... the term “union” is poor' 
> statement. Are there some better alternative terms, perhaps 
> 'multi-index operator', 'index list', 'subscript list', etc.?
>
> ## Duplicates and Ordering
>
> > It was my impression that we were talking about duplicated nodes not
> duplicated values:
> >
> > Given th array [10,20,30]
> >
> > $..[0,1,0]
> >
> > Would yield only two results [10, 20]
> >
> > (Not that I'm advocating for removing duplicates, personally I think we
> shouldn't)
> *(Marko Mikulicic)*
>
> > You’re framing this as “removing duplicates”.
> Another view is that [10, 20, 10] would be “adding duplicates” (copies 
> of the same node). Related are ordering issues:
> >
> > `$..[1,0] ➔ [20, 10] Or [10, 20]`
> >
> > I would expect the spec will leaves implementations some leeway 
> here, but that should be based on an examination of existing 
> implementations.
> *(Carsten Bormann)*
>
> > The mental model that leads to omitting duplicate nodes in the 
> output is
> "selection": if you take an input array and select nodes with index 
> 0,1 or
> 0, you get only 2 results (since selecting an index twice has no effect).
> >
> >OTOH, if you opt for a "collect" model, whenever you encounter a node 
> that
> matches that query you add it to the result stream, thus the same 
> nodes can
> be present multiple times in the result.
> >
> >I have a slight preference for the "collect" model, because the general
> case in jsonpath is to collect things that appear at various points in 
> the
> json tree. For example:
> >
> >`{"a": {"b": 1, "c": 2}, "d": 3},  $.a.b yields [1] and not 
> {"a":{"b":1}}`
> >
> >(i.e. jsonpath is not a filter and view operation but a pick and gather
> operation)
> *(Marko Mikulicic)*
>
> > In implementations that support paths (the majority don't), the query
> function takes a parameter that indicates values or paths. In both
> cases the query returns a JSON array of JSON values, in the latter
> case, a JSON array of normalized paths.
> *(Daniel P)*
>
> I must confess to never having thought about duplicates, let alone 
> wanting to eliminate them. So I do like Marko's comparison of 
> 'selection-model' vs. 'collection-model' a lot. I would opt for the 
> latter. In this sense the result of a 'JSONPath query expression' 
> should be termed a 'collection'.
>
> Regarding ordering I see something like a 'natural ordering', 
> according to which
>
> `$..[0,1] ➔ [10, 20]`
> `$..[1,0] ➔ [20, 10]`
>
> would result with the example above.
>
> I do understand the use cases for reordering, duplicates removal, 
> filtering, etc.. But this can always be seen as a postprocessing step 
> on the resulting collection by handing it over to accompanying tools 
> (think of pipe operator).
>
> Of course this cannot work on the result collection of values alone 
> (s. duplicate nodes vs. duplicate values above), it rather requires a 
> collection of (normalized) pathes. In this sense, I like this view:
>
> > In my opinion the right balance between powerfulness and enabling
> simple implementations has been so far one of the key factors that
> made JSONPath popular over other alternatives, even if it lacks
> support for aggregation functions.
> *(Davide Bettio)*
>
> ## Filter Expressions
>
> > Related to that, it would be helpful to determine if JSONPath filters
> apply to both JSON objects and arrays, or only to JSON arrays.
> *(Daniel P)*
>
> > I would support restricting filters to arrays, if others agree.
> *(Glyn Normington)*
>
> I tend to let implementations and their "normative force of the 
> factual" decide here or in doubt agree to Glyn's restriction to arrays.
>
> I am very unhappy with confusing `$..book[(@.length-1)]`, where `'@'` 
> addresses the array itself and implies that array has a `length` 
> property. In filter expression examples `'@'` more consistently 
> addresses the current array element.
>
> The invocation of 'the underlying scripting engine' wasn't meant a 
> serious normative aspect, but rather a quick and dirty solution for 
> JavaScript and PHP implementations at that time.
>
>
> ### Corner Case
>
> > Consider this perfectly legal JSON object
> >
> > ```{ "ab": 0,  "'a.b": 1,  "a-b": 2, "a": { "b": 3 } }```
> >
> >So `$.ab` is 0, `$.a.b` is 3, `$['a.b']` is 1, `$['a-b']` is 2. You'd 
> like to say `$.a-b` but lots of libraries will refuse it because 
> `"a-b"` is not a legal JavaScript "name" construct, that's why you 
> have to say `$['a-b']`.
> >
> > But suppose your library would accept `$.a-b`.  Then `$.a-b` and 
> `$['a-b']` would be synonyms, but `$.a.b` and `$['a.b']` wouldn't.
> *(Tim Bray)*
>
> Hmm ... this seems to be a hint to better exclude `'-'` from 
> dot-child-selector syntax. I think I have read more discussion about 
> that, currently don't know where.
>
> ## Respect Implementations
>
> > As I mentioned in the session, I think there's a non-trivial amount 
> of risk here that some implementations won't be willing or able to 
> move away from their current behaviours, even if interoperability 
> would improve if they did so. However, there are ways to mitigate that 
> (e.g., a separate 'rfcxxxx compliant' mode). Even so, it will be 
> important to get good participation from as many current implementers 
> as possible.
> *(Mark Nottingham)*
>
> > The WG will develop a standards-track JSONPath specification that
> is technically sound and complete, based on the common semantics
> and other aspects of existing implementations.  Where there are
> differences, the working group will analyze those differences and
> make choices that rough consensus considers technically best, with
> an aim toward minimizing disruption among the different JSONPath
> implementations.
> *(Barry Leiba)*
>
> > I'm OK with this, but for context: I've been a pretty intense 
> JSONPath user
> in recent years, and AFAIK the spec, and the implementations, are mostly
> OK, so the choice between "make JSONPath good" and "don't invalidate
> implementations" is unlikely to come up. If it did, my predisposition 
> would
> be to err on the side of not breaking implementations, but I don't think
> that's inconsistent with Barry's text.
> *(Tim Bray)*
>
> +1 to all.
>
> ## Error Handling
>
> > My mental model at the moment is that a JSONPath expression can be 
> valid or erroneous; application of a valid expression yields a result 
> (which may be empty), but does not raise errors. That may not be the 
> right model for all applications.
> *(Carsten Bormann)*
>
> > The  general approach that I've seen several times (including my
> Elixir implementation) is that an error is raised when there is a
> syntax error, therefore an invalid expression (e.g. $.foo[[5]) raises
> an error. Conversely a valid expression applied to a bogus input never
> raises an error (e.g. `$.foo.bar on "test" evals as []`).
> *(Davide Bettio)*
>
> > On the whole I think JSONPath is designed to be "forgiving", i.e. 
> such things aren't errors, e.g. I think I read in the spec that 
> filtering a non-array isn't an error, it's some kind of no-op. That 
> approach isn't always best for everyone, but it's important to be 
> consistent.
> *(Michael Kay)*
>
> > I would expect one component of this policy to be:
> >
> > Whether a JSONPath query is valid or not does not depend on the 
> arguments it is applied to.
> >
> > I.e., you can look at the query and find out independently, without 
> knowing any data, whether it is valid or not.
> *(Carsten Bormann)*
>
> I like and totally agree with the *forgiving mental model*, so having  
> only syntax errors, which do not dependent on data.
>
> Thanks
> -- 
> sg