Re: [Jmap] new JMAP server for prototyping

[Neil Jenkins <[e-mail]>]

Hi Jamey,

> Hey all: for the past few weeks I've been working on a new JMAP server with the goal of making it easy to prototype new data models.

That's awesome! Nice work.

> I've been confused by a few things in RFC8620. I was only going to give a couple examples but ended up with a complete list of everything I remember having trouble with, so I hope it works out to just dump it here. I have some more notes afterward on my current implementation and plans.
> 
> - Should creating multiple objects allow circular references, or must it be a DAG of new objects? The former seems hard to get right if any of the creations fail.

The relevant bit of the spec here is:

   The final state MUST be valid after the "Foo/set" is finished;
   however, the server may have to transition through invalid
   intermediate states (not exposed to the client) while processing the
   individual create/update/destroy requests.  For example, suppose
   there is a "name" property that must be unique.  A single method call
   could rename an object A => B and simultaneously rename another
   object B => A.  If the final state is valid, this is allowed.
   Otherwise, each creation, modification, or destruction of an object
   should be processed sequentially and accepted/rejected based on the
   current server state.

I don't think we actually considered a data type that supported circular references, but the way the current spec is written I would say that if the circular reference is valid for that data type and the final state is valid, then yes you would be allowed to do that in a single /set. But I agree that's a bit ambiguous, and I think you could probably forbid it in the /set description for the particular data type. Do you have an example in mind?

> - Can result references appear in nested objects within a method's arguments, or only on the arguments directly within the top-level Invocation?

Only on the top-level. The *arguments object* is the top-level object passed in the *Invocation*. So this:

   When processing a method call, the server MUST first check
   the arguments object for any names beginning with "#".

means just the top-level object.

> - Does the client need to ensure that it never uses a creation ID of "foo" if there's some random string whose value happens to be "#foo", or does /set need to know which properties have Id type?

When processing a /set you need to know which properties have Id type (well you need to know all the types really in order to be able to validate the data).

> - If the client uses the same creation ID in two method calls, despite the "SHOULD" cautioning against it, and the second create fails, should the first one's ID continue to be used? Is "the most recently created item with that id" the most recent attempt, or the most recent success?

Most recent success was the intention, but looking at it again I can see how you could interpret it both ways.

> - I think I get it now, but I've been confused about how the responses from /copy relate to its three phases. There's one /copy response covering phases 1 and 2, then optionally a single /set response for phase 3, right?

That's right.

> - What value should the position attribute in a /query response have if the requested position is past the end of the results?

The definition of "position" in the response is:

      The zero-based index of the first result in the "ids" array within
      the complete list of query results.

So I agree it's undefined if there are no results (but I also think it's unimportant, because there are no results to position!). I would just return the requested position I think. It would probably be worth making this well-defined in a future revision.

> - If a client has a sparse array of query results and gets a /queryChanges response indicating that it should delete an object that it didn't have in the array, then I don't see how it can tell whether to shift down any of the objects it does have: the deleted object could have been anywhere. This interface only seems safe so long as clients either always keep a complete prefix of a query's results, or invalidate their cache if told to delete an object they didn't know about.

Close. In this situation you have to invalidate any results in your sparse query after the first gap, but you can still keep the ones before. So if I have:

[A, B, -, -, D, -, -]

and queryChanges tells me that E has been removed, I would have to invalidate after the first gap, so my cached query would now be:

[A, B, -, -. -. -. -]

> - What does property immutability have to do with query result changes? Isn't the only thing that matters whether the property actually changed, rather than whether it could change? Is it just that some reasonable implementations can't compute certain changes for mutable properties?

Yes, it's that the spec was written with the idea that /queryChanges would be implemented based on knowing which records have changed from last time, not based on having the old query cached and comparing it with the new one. (This approach is hard to scale effectively.) If you only know that a record has changed, not which property on that record then you don't know what position it had in the client's current query cache (unless you have cached the old query, as you are doing), so have to tell the client to remove it an readd it to ensure it's correct.

If you can return precise changes with your implementation, that's great! This is really about guidance for what to do when you can calculate usable, if slightly less concise, changes efficiently. The text could definitely be better here, it's written a bit too much with an implementation approach in mind (in general we tried to not do that, as there are multiple good ways of implementing various bits of JMAP).

> - The specification for upToId seems to imply to look up the position of the given ID in the new results, though that isn't entirely clear to me. 

Yes, in the new results.

> But I think it makes more sense to find its position in the old results, and send changes that update the client's cache to the same length. That means the optimization still works even if the selected object is no longer in the results, and puts an upper bound on how much data the client might have to deal with: twice the number of items it has cached already.

I'm not quite sure what you mean by "to the same length", but yes if you have the old query cached you could optimise the case where the upToId is not in the new results a bit further. But again, the spec is written to allow implementers to calculate query changes *without* caching the old query state.

> But overall I've found the spec pretty clear and solid so far.

Great! This was really good feedback. I don't think any of it requires an errata (although if someone on the list disagrees, please pipe up!), but if we publish an updated RFC at some point in the future we know which points to clarify.

> Currently all my tests are property-based randomized testing. I'd like to think my tests might be interesting for other server implementors. If there are test suites I might be able to reuse, I'd love to hear about them.

Getting a good test suite for the JMAP spec is something we would love to have (and know from past experience is important for interoperability). I don't think anyone has published an open reusable one yet.

> I've yet to tackle the session state, blobs, or push, but those don't seem like they'll have hard questions about semantics, just some possibly tricky engineering choices. We'll see how that goes.

I think queryChanges is the trickiest bit to get your head around semantically so hopefully now that's done the rest is not too hard, but do keep us informed of your progress and any other spec issues you find!

Cheers,
Neil.