Subject: Re: [imapext] [ietf-smtp] Fwd: Request to form a new WG: JMAP
From: Neil Jenkins <[email-masked]>
To: [email-masked]
Date: Fri, 11 Nov 2016 14:57:45 +1100
List-Id: Discussion of IMAP extensions <imapext.ietf.org>

Hi all,

Rather than reply to individual emails, I'm going to try to put across
answers to the queries and concerns related so far in this thread. I'm
the author of the draft specification we're putting forward with this
charter; happy to answer any further questions.

# What is JMAP replacing?

JMAP is intended to be a new standard for email clients to connect to
mail stores. It therefore replaces IMAP + SMTP submission. It is also
designed to be more generic such that it can be extended with contacts,
calendars in the future (replacing CardDAV/CalDAV). It does not replace
MTA-to-MTA SMTP transmission.

# Why is this needed?

It's too hard to write a good MUA with current standards, which has led
to a stagnation in good email clients and a proliferation of proprietary
protocols.

In addition, IMAP is really not suited to a constrained network
environment, as often found on mobile networks even in developed
countries. The chatty nature of the IMAP protocol is ill-suited to high
latency connections. Things like a folder rename on the server can mean
that all the downloaded and cached email for that folder is invalidated
(because you can't tell for sure over the protocol that they're the
exact same messages), wasting huge amounts of bandwidth. Stateful
connections make it harder to deal with intermittent network outages.

We're attempting to standardise a new protocol because lots of people
are writing proprietary alternatives to deal with the same deficiencies
with the current standards. Some of these deficiencies could be fixed by
adding things to IMAP (for example persistent IDs on folders and
messages for better caching when another client copies, moves or
renames), but others are structural like the need to calculate a MSGNO
<=> UID mapping even if the client doesn't want it, or the need to find
endpoints and authenticate separately to multiple different protocols to
do related tasks.

Because it's so hard to write a good email client that works with any
IMAP provider, these days many new clients are just for Gmail (with
possibly support for a few other big mailbox providers). It's easy to
see why: IMAP is either the woefully inadequate original RFC3501, or
it's a messy set of incomplete implementations of some of the
extensions. Even CONDSTORE (without QRESYNC) is pushing it; you can't
rely on it to be there in most cases.

As a result of this, proprietary protocols have been popping up as
alternatives to IMAP. Here are a few examples, the latter two being from
whole companies that formed just to try to help people not have to deal
with IMAP:

Gmail: https://developers.google.com/gmail/api/v1/reference/
Outlook:
https://msdn.microsoft.com/office/office365/APi/mail-rest-operations
Nylas: https://nylas.com/cloud/docs
Context.io: https://context.io/docs/lite

In addition, we're seeing most new mobile email clients proxy everything
via their own server rather than talking directly to the user's mail
store, for much the same reasons. Examples include Mailbox (now
defunct), Alto, Outlook and Newton. This is bad for security and
privacy, and also bad for the client authors as they have to run server
infrastructure in addition to just building their clients.

Despite not only being proprietary but patented (and expensive!),
ActiveSync has seen a big increase in adoption, and not just with
Microsoft servers, due to its better support for mobile environments and
ease of setup (one login for mail receive/send, contacts and calendars).

# Why is JMAP better than IMAP?

JMAP not a conversion of IMAP to JSON; it is a new protocol. It was
designed to be make much more efficient use of network resources, be
easier for developers to work with and hopefully make the best
protocol for email an open standard once more. It's based on years of
experience and real-world experimentation at FastMail, and talking to
other major MUA/MTA developers to make sure we understand the common
needs of the industry.

Some important attributes that help achieve these goals:

* The protocol is stateless. It doesn't need a persistent connection,
  which is better for mobile use which may have intermittent network
  access and wants to conserve battery life by turning the radio off
  whenever possible.

* Ids are immutable and not intended to be user visible. So folder
  naming becomes less messy - more like NFS or filesystems with inodes
  rather than a name-based hierarchy, and renaming is easy to detect and
  cheap to sync.

* It has a flexible set of commands, which can be batched in arbitrary
  ways.  You can batch or pipeline single JMAP operations over a stream
  protocol easily enough if you wanted to, but we're mostly envisaging
  it being used for stateless batch operations to make disconnection
  less painful.

With IMAP you can set two messages to both have the same flag (. STORE
1,2 +FLAGS (aflag)) but you can't store two different flags to two
different messages in the same action. JMAP allows multiple create,
update and destroy actions on different messages in a single setMessages
command. Pipelining also has the problem that if the connection drops at
just the wrong moment you can wind up applying the first change but not
the second.

You can use backreferences to other objects created in the same batch -
allowing you to, for example, create a folder tree by referencing
previous parents created in the same request.

* Clients can efficiently fetch updates from their current state a-la
  QRESYNC. This can be implemented effectively using the MODSEQ data
  already in modern IMAP servers, or by using a transaction log data
  structure. The server can always indicate to the client if it
  cannot calculate updates from a particular client state (e.g.
  because it is too old).

* Flood control. The client can always restrict how much data the server
  should send. For example, a command might return a "tooManyUpdates"
  error if it exceeds the client's limit, rather than returning a
  million "* 1 EXPUNGED" lines as can happen in IMAP. Sometimes it's
  just more efficient to throw away cached data and refetch, especially
  if you're a mobile/webmail interface with only a partial cache of the
  server's data.

* It doesn't require a custom parser. I've got a longer explanation to
  the HTTPS/JSON question lower down, but having an encoding format that
  is well understood and has widespread support among all programming
  languages makes it far easier for developers to get started,
  especially if they don't want to build a whole MUA but just integrate
  something with email.

* The data model is backwards compatible with both IMAP folders and
gmail-
  style labels. Servers that implement JMAP are likely to want to
  support IMAP as well for the foreseeable future, so it's important to
  be able to have data structures that support both. Messages are
  similarly immutable other than flags/mailboxes.

* Email can be sent using the same protocol, reducing confusing failure
  modes for users (again I talk more about this below). We also have
  pretty-
  much complete specs for calendaring and contacts via JMAP, but we're
  not pushing for them to be standard yet because the object format is
  still undergoing a lot of work in the CalConnect group. We think a
  single consistent protocol for all of these has a lot of advantages
  though, and we hope to get there in the future.

# Why use HTTPS/JSON?

The short answer is it's good enough, widely understood and it's by far
the easiest thing for developers to adopt. There's support in basically
all OSes and programming languages. It's easy to read and debug.

HTTP doesn't tend to run into firewall issues, and is so commonly used
it has integrations which can help with optimisation (for example, iOS
has built-in support for optimising radio usage by batching HTTP calls
from different apps where possible, which their mail team have told us
they would like to be able to use). This isn't an innate advantage of
HTTP, but rather an advantage of its ubiquity.

With GZIP, JSON data is reasonably compact and fast enough to
serialise/parse. However, the encoding/transport part of JMAP is not
core to its operation, so future specifications could easily add
alternatives (e.g. WebSocket (https://tools.ietf.org/html/rfc6455)
instead of HTTPS, CBOR (http://tools.ietf.org/html/rfc7049) instead of
JSON). For the initial version though, HTTPS+JSON makes the most sense.

# Binary data

Binary data is not transported in the JSON (and indeed, as has been
pointed out, can't be without base64 encoding or similar, which is
inefficient). Instead, attachments are referenced by a blobId, and
uploaded/downloaded separately via HTTPS. Clients can reference the
blobId elsewhere to, for example, attach the same file to a new message
without having to download and reupload it again, a big win on slower
internet connections.

This also means that regularly saving drafts (a common client behaviour)
does not mean sending the same full multi-megabyte attachments over the
network every 60s or so.

As it's out-of-band with the API calls, uploading/downloading files can
easily be parallelised and doesn't block other API operations.

# Representation of email

JMAP defines a JSON structure that represents in a consistent and
structured way all the information that the vast majority of
clients need from an RFC5322 message. The server deals with the
complexities of MIME, encoding issues, parsing headers etc. The
intention is that the server will still operate with RFC5322
messages for storage and certainly transmission; the JSON
representation is not intended to replace RFC5322, just relieve
client authors from having to deal with it.

Clients that want to or need to (for example those doing PGP in the
client) can still fetch the RFC5322 if needed. The message is
represented by a blobId, and the raw bytes can be fetched using the same
binary download mechanism as mentioned above.

# Message submission

Message submission is via means of an "outbox" folder. Messages are
moved there to send. This was chosen over a separate "send" command for
a few reasons. Firstly it's most consist with the rest of the API,
making it easier for clients to implement offline support
(synchronisation is the same as other changes you might make to
messages). Secondly, clients/servers can support delayed send
(particularly useful for "undo send"), by simply setting the date on the
message in the future and only sending from the outbox when this date is
reached. Until then the message is just sitting in the outbox like any
other mailbox, allowing client to list and revoke without needing custom
API commands.

Error handling is flexible enough to return a full range of errors when
you try to move a message to this folder, just as you would to a
separate "send" method.

Clients can use the same JSON structure for sending messages as they get
from the server for received messages, allowing the server to deal with
MIME encoding. This allows clients to be much simpler and easier to
write. (They can also upload a raw RFC5322 message if they want instead
of course.)

Having the same protocol for message sync and submission is a huge win
for usability; we see a lot of support tickets where users can receive
but not send, or vice versa, because one of these is misconfigured. This
is always very confusing for regular users.

# Push mechanism

Immediate updates is an important feature to many users. IMAP IDLE has
two big problems: firstly it only notifies of changes in one folder, so
doesn't inform you of all changes unless you open a connection for every
folder, and secondly it requires a persistent network connection which
is bad for mobile (and not even allowed on iOS).

JMAP defines two push mechanisms to support the two common use cases. In
both cases the data transferred is simply an edge trigger: a new state
string letting the client know something has changed within a particular
datatype. The client then fetches the new data using the standard
synchronisation methods.

For desktop clients and webmail, there's an event source interface
(https://html.spec.whatwg.org/multipage/comms.html#the-eventsource-interface)
. This requires a persistent HTTP connection.

For mobile, and web integrations, you can set a callback handler. This
makes the mail store server do a callback to a server defined by the
client when something changes; the client's server can then send out-of-
band push events using the native push mechanism of the mobile client.
JMAP itself doesn't require any particular mobile push technology.

# End-to-end encryption

A lot of the optimisations for efficient client-server sync require the
server to be able to read the message. If everything were encrypted, the
server would basically be a dumb blob store, unless you have some clever
partial metadata search capability. This is particularly bad for mobile,
where you only want to sync partial information. Users expect to be able
to search their whole archive, so either you need all the data in the
client, or the server needs to have access to the data.

JMAP is therefore not making any new measures to address end-to-end
encryption. The best advice is probably to run your own "JMAP SERVER" on
trusted hardware; otherwise you need to sync the entire multi-gigabyte
mail spool to all your devices. JMAP is also simple enough that you
could run the server on multiple machines with an underlying replication
protocol over encrypted links and have that do your smarts.

# Draft proposals and implementations

As Arnt pointed out, "rough consensus and running code" are key here,
and the current draft of the JMAP spec is being implemented in Cyrus
IMAPd and Dovecot (the two largest open-source IMAP servers), as well as
other open source projects like Apache James and Linagora. Roundcube
have stated they plan to build their next-generation client on JMAP.

On the proprietary side, Atmail have a JMAP proxy and webmail client
they are releasing to production very soon, and we at FastMail have a
version of our client that talks JMAP too.

There is interest among other large mailbox providers/mail client
authors, but they don't tend to be early adopters as much.

The current specification drafts can be found at:

https://datatracker.ietf.org/doc/draft-jenkins-jmap/ (The generic JMAP
protocol)
https://datatracker.ietf.org/doc/draft-jenkins-jmapmail/ (Mail over
JMAP)

We have also developed an open-source JMAP proxy, which you can connect
to an IMAP server to try out the protocol and see what happens over the
wire. There's a hosted version at http://proxy.jmap.io/, or you can grab
the code from https://github.com/jmapio/jmap-perl

Cheers,

Neil.