[Jmap] JMAP Conceptual Decisions

[Neil Jenkins <[e-mail]>]

The core JMAP protocol has a number of conceptual decisions. Here are
the choices we made in the initial design and the reasons behind them,
so it's all documented on this mailing list.
The core conceptual decisions, I think, are:
 * Use of HTTP + JSON
 * Separate end points for binary data upload/download
 * Concurrency model
 * Structure of request/response as array of [ method-name, data,
   tag ] tuples
Why use HTTPS/JSON?


The short answer is it's good enough, widely understood and it's by far
the easiest thing for developers to adopt. There's support in basically
all OSes and programming languages. It's easy to read and debug.HTTP doesn't tend to run into firewall issues, and is so commonly used
it has integrations which can help with optimisation (for example, iOS
has built-in support for optimising radio usage by batching HTTP calls
from different apps where possible, which Apple's mail team have said
they would like to be able to use). This isn't an innate advantage of
HTTP, but rather an advantage of its ubiquity.With GZIP, JSON data is reasonably compact and fast enough to
serialise/parse. However, the encoding/transport part of JMAP is not
core to its operation, so future specifications could easily add
alternatives (e.g. WebSocket[1] instead of HTTPS, CBOR[2] instead of
JSON). For the initial version though, HTTPS+JSON makes the most sense.Handling of binary data


Binary data is not transported in the JSON as it can't be without base64
encoding or similar, which is inefficient. Instead, binary blobs are
always referenced by a blobId, and uploaded/downloaded separately via
HTTPS. As it's out-of-band with the API calls, uploading/downloading
files can easily be parallelised and doesn't block other API operations,
so a client can remain responsive and make new API requests while
uploading, for example, a large attachment concurrently.Clients can also reference the blobId elsewhere to, for example, attach
the same file to a new message without having to download and reupload
it again, a big win on slower internet connections. This also means that
regularly saving drafts (a common client behaviour) does not mean
sending the same full multi-megabyte attachments over the network every
60s or so, as happens with IMAP.Concurrency model


Straight forward but worth mentioning. To ensure the client always sees
a consistent view of the data, the state accessed by a method call MUST
NOT change during the execution of the method, except due to actions by
the method call itself.Concurrent requests are allowed (subject to a server-defined limit), but
the server must ensure that the above constraint is not violated. This
is basically the minimum required for sane behaviour, while still giving
the server wide flexibility in implementation.Structure of request/response


JMAP does not use HTTP verbs for the API requests (everything goes over
a POST request to a single end point). This decision was made primarily
for efficiency, which is best illustrated by example:Suppose you are looking at your inbox, and decide you want to move a
message into a new mailbox which you will call "Invoices 2017". The
client also needs to get the changes this results in to our query
listing the ids of threads in the Inbox. I don't believe this is
contrived; this functionality exists in many clients today.With the JMAP model this is one HTTP request:


POST /jmap-api/ [ [ "setMailboxes", { "create": { "clientId": { "name":
"Invoices 2017" ... other properties elided ... } } }, "tag-0" ], [
"setMessages", { "update": { "msg123": { "mailboxes": [ "#clientId" ] }
} }, "tag-1"], [ "getMessageListUpdates", { "filter": { "inMailbox":
"inboxId" }, "sort": [ "date desc" ], "collapseThreads": true,
"sinceState": "deadbeef", "maxChanges": 50 }, "tag-2"] ]which would return something like:


200 OK [ [ "mailboxesSet", { "accountId": "foo", "oldState": "x1234",
"newState" "x1237", "created": { "clientId": { "id": "mbox98" ... other
server-set properties elided ... } } }, "tag-0" ], [ "messagesSet", {
"accountId": "foo", "oldState": "mx987", "newState" "mx7141", "updated":
{ "msg123": null } }, "tag-1"], [ "messageListUpdates", { "accountId":
"foo", "filter": { "inMailbox": "inboxId" }, "sort": [ "date desc" ],
"collapseThreads": true, "oldState": "deadbeef", "newState": "12345678",
"total": 587120, "removed": [ { "messageId": "msg123", "threadId":
"thd17" } ], "added": [] }, "tag-2"] ]This example illustrates a few things:


 * Arbitrary bundling of methods into a single HTTP request lets you
   minimise round trips, while being flexible and easy to implement.
 * Methods are processed in order, so you have a guarantee that the
   message will be modified before the message list updates are
   calculated.
 * Use of back references (the message is set to move to mailbox with id
   #clientId – this is not the real, server-assigned id, but the id the
   client gave the server to keep track of it for the duration of this
   request. It is trivial for the server to keep a map of clientId ->
   real id for each record it creates for the duration of HTTP request
   (it can then throw this away; it does not need to persist it).
   Forward references not allowed!
 * The third item in the tuple for each method request is an arbitrary
   "tag" given by the client, which is just echoed back in the response.
   Although none do in this example, a method may return multiple
   responses, so this allows them to be correlated with requests if
   necessary. (Note a lot of the time you may not even need this: our
   client mainly just handles the responses based on the "name" of the
   response; it doesn't care who called or why).Now, let's look at what this same exchange might look like with a more
traditional REST style:


POST /mailboxes/ { "name": "Invoices 2017" ... other properties
elided ... }which might return:


201 Created { "id": "mbox98" }
and then you can do:


PATCH /messages/msg123 { "mailboxes": [ "mbox98" ] }wait for the response, and finally do


GET /messagelist/updates?filter={"inMailbox":"inboxId"}&sort=datedesc&c-
ollapsethreads=true&sinceState=deadbeef&maxChanges=50(Yeh, not sure about that one; I don't think many REST APIs have a
concept of delta updates to query results.)We needed three round trips to do the same thing. When round trip times
are 300ms or more (such as Australia talking to East Coast US, or just
many cellular connections), this is a really noticeable performance hit.With HTTP/2 you could maybe use dependent streams to parallelise the
latter two requests, but this is more complicated for both client and
server authors, and has far less support out there (as far as I know you
couldn't do this from a browser for example). You still have to wait for
the first request to complete even with dependent streams, because you
can't do the backreferences (if you did, how long does the server have
to hold on to them? Forever? There's no neat method bundle like with the
JMAP API request format).This example didn't even do batching of changes to multiple records.
Suppose you wanted to move 10 messages to that new folder, it's still 3
method calls and 1 HTTP request in JMAP. With HTTP you now have an extra
9 requests. Even with HTTP/2 to reduce the transport overhead, you would
still have to work much harder to make the server efficient (you will
need to take out locks on some data before you can make the changes: for
efficiency you want to do this once and modify all 9 items then release
the locks).I hope I have illustrated the reasoning behind the format we chose
when designing the current JMAP draft. Compared to HTTP REST, I
believe it is:
 * More efficient
 * More consistent (can do batching, or non standard CRUD such as
   importMessages/copyMessages methods, all in the same style)
 * Just as easy to use, if a little unfamiliar at first
 * Not tied to HTTP (as mentioned above, WebSockets or other future
   transport may be desired in the future).
Neil.

Links:

  1. https://tools.ietf.org/html/rfc6455
  2. http://tools.ietf.org/html/rfc7049