Re: [Jmap] new JMAP server for prototyping

[Neil Jenkins <[e-mail]>]

On Sat, 22 May 2021, at 06:20, Jamey Sharp wrote:
> Looking at a different bit of the text:
> 
>     In the case of records with references to the same type, the server 
>     MUST order the creates and updates within a single method call so 
>     that creates happen before their creation ids are referenced by 
>     another create/update/destroy in the same call.
> 
> I think this wording is a little strange because I don't think destroys 
> can reference creation IDs at all, right?

Technically it could destroy a creation in the same request (it's a weird edge case, but it was worded that way deliberately); I know we actually came across this scenario at some point, although I can't remember why just now.

> But more importantly: I think a strict reading would prohibit circular 
> references in creates, since there is no ordering satisfying this MUST 
> in that case.

Yes, I would read that the same; I think this paragraph does implicitly rule out creating circular references directly.

> It seems easy enough for a client to use random creation Ids which are 
> sufficiently large that they never collide with each other or with 
> user-provided data. So if it's intended that clients may use strings 
> which match their own creation Ids, I might just decide this 
> implementation doesn't interoperate with clients that do that, which 
> seems okay in a prototyping setting.

Hmm, I mean, sure you can do whatever you want for prototyping, but this is definitely not spec compliant. (My client, for example, just uses an incrementing integer for creation ids, which avoids reuse as per spec but is definitely not guaranteed to not appear by coincidence in other properties).

> "Most recent success" means a client which violates this 
> "SHOULD" can't predict which object it's going to be referencing, and 
> the target could even be of a different type than expected. But after 
> all, you did tell them they shouldn't do that…

It's a SHOULD because the server behaviour is (in theory) well defined, and in theory you could do something crazy (e.g. Try creating three different items then set a reference to the last one that succeeded…).

> >   The definition of "position" in the response is
> 
> I agree it shouldn't matter, but I think it might be helpful to define 
> it any of these equivalent ways:
> 
> - the position immediately after the last item in the list
> - the length of the list
> - the minimum value of position for which you'd get the same response

Sure, I think this value is a reasonable choice. It does mean there are a few optimisations you can't make though (because it's essentially the same as forcing the server to calculate the total), although they are unlikely to be important ones I think.

> >   Close. In this situation you have to invalidate any results in your 
> >   sparse query after the first gap, but you can still keep the ones 
> >   before.
> 
> Ohhhh. That makes sense, thanks. The spec is wrong here though, I think:
> 
>     The result of this is that if the client has a cached sparse array of
>     Foo ids corresponding to the results in the old state, then:
> 
>     fooIds = [ "id1", "id2", null, null, "id3", "id4", null, null, null ]
> 
>     If it *splices out* all ids in the removed array that it has in its
>     cached results, then:
> 
>        removed = [ "id2", "id31", ... ];
>        fooIds => [ "id1", null, null, "id3", "id4", null, null, null ]
> 
>     and *splices in* (one by one in order, starting with the lowest
>     index) all of the ids in the added array:
> 
>    added = [{ id: "id5", index: 0, ... }];
>    fooIds => [ "id5", "id1", null, null, "id3", "id4", null, null, null ]
> 
> Here, "id31" is shown as removed, which wasn't in fooIds, but "id3" and 
> "id4" come after a gap and weren't removed.

Hey damn, you're right! I think my brain glossed over this when proofreading because the line below starts **and truncates**, and in my code the flag that gets set when you have an unknown destroyed id is called `truncateAtFirstGap`. But the spec is actually just description the final step of adjusting the query results length, not removing the ids after the gap.

Would you like to submit the errata? Or I'm happy to do it.

> >   I'm not quite sure what you mean by "to the same length", but yes if 
> >   you have the old query cached you could optimise the case where the 
> >   upToId is not in the new results a bit further. But again, the spec 
> >   is written to allow implementers to calculate query changes without 
> >   caching the old query state.
> 
> I meant that if the client has cached the first 30 items, then upToId 
> refers to the 30th item they knew about; a delta which gets them back to 
> 30 cached items seems like a reasonable thing to want.
> […]
> I can't quite figure out what client UX the upToId feature supports. I 
> can imagine trying to keep the ten threads starting at #23 visible on 
> the screen, or keep the thread list centered around the thread which is 
> currently being displayed, but there's no reason to expect this will let 
> me do either of those things, right?

It's purely an optimisation in both what's sent over the wire and (for certain implementations) how much work the server does. In most scenarios, a client will have just the start of a list cached. Extreme scenario, we have the first 3 ids cached from a list of 500 results:

[ id1, id2, id3, null4 … null500 ]

The goal of queryChanges is to ensure that the state of the client after applying the changes is an accurate representation of the new state on the server, while sending a minimal delta. Suppose the last 100 items are removed, and a further 50 items are inserted at the end instead. Simply truncating the client's sparse array to the new length (450) means it is now a valid representation of the new state: every id that's there is in the correct position. This is what upToId does: instead of sending 50 "added" and 100 "removed" ids you can just send the new total instead.

Cheers,
Neil.