[Jmap] Fwd: Request to form a new WG: JMAP

[Alexey Melnikov <[e-mail]>]

Begin forwarded message:

> From: Neil Jenkins <[e-mail]>
> Date: 11 November 2016 at 12:57:45 GMT+9
> To: [e-mail]
> Subject: Re: [imapext] [ietf-smtp] Fwd: Request to form a new WG: JMAP
> 
> Hi all,
> 
> Rather than reply to individual emails, I'm going to try to put across
> answers to the queries and concerns related so far in this thread. I'm
> the author of the draft specification we're putting forward with this
> charter; happy to answer any further questions.
> 
> # What is JMAP replacing?
> 
> JMAP is intended to be a new standard for email clients to connect to
> mail stores. It therefore replaces IMAP + SMTP submission. It is also
> designed to be more generic such that it can be extended with contacts,
> calendars in the future (replacing CardDAV/CalDAV). It does not replace
> MTA-to-MTA SMTP transmission.
> 
> # Why is this needed?
> 
> It's too hard to write a good MUA with current standards, which has led
> to a stagnation in good email clients and a proliferation of proprietary
> protocols.
> 
> In addition, IMAP is really not suited to a constrained network
> environment, as often found on mobile networks even in developed
> countries. The chatty nature of the IMAP protocol is ill-suited to high
> latency connections. Things like a folder rename on the server can mean
> that all the downloaded and cached email for that folder is invalidated
> (because you can't tell for sure over the protocol that they're the
> exact same messages), wasting huge amounts of bandwidth. Stateful
> connections make it harder to deal with intermittent network outages.
> 
> We're attempting to standardise a new protocol because lots of people
> are writing proprietary alternatives to deal with the same deficiencies
> with the current standards. Some of these deficiencies could be fixed by
> adding things to IMAP (for example persistent IDs on folders and
> messages for better caching when another client copies, moves or
> renames), but others are structural like the need to calculate a MSGNO
> <=> UID mapping even if the client doesn't want it, or the need to find
> endpoints and authenticate separately to multiple different protocols to
> do related tasks.
> 
> Because it's so hard to write a good email client that works with any
> IMAP provider, these days many new clients are just for Gmail (with
> possibly support for a few other big mailbox providers). It's easy to
> see why: IMAP is either the woefully inadequate original RFC3501, or
> it's a messy set of incomplete implementations of some of the
> extensions. Even CONDSTORE (without QRESYNC) is pushing it; you can't
> rely on it to be there in most cases.
> 
> As a result of this, proprietary protocols have been popping up as
> alternatives to IMAP. Here are a few examples, the latter two being from
> whole companies that formed just to try to help people not have to deal
> with IMAP:
> 
> Gmail: https://developers.google.com/gmail/api/v1/reference/
> Outlook:
> https://msdn.microsoft.com/office/office365/APi/mail-rest-operations
> Nylas: https://nylas.com/cloud/docs
> Context.io: https://context.io/docs/lite
> 
> In addition, we're seeing most new mobile email clients proxy everything
> via their own server rather than talking directly to the user's mail
> store, for much the same reasons. Examples include Mailbox (now
> defunct), Alto, Outlook and Newton. This is bad for security and
> privacy, and also bad for the client authors as they have to run server
> infrastructure in addition to just building their clients.
> 
> Despite not only being proprietary but patented (and expensive!),
> ActiveSync has seen a big increase in adoption, and not just with
> Microsoft servers, due to its better support for mobile environments and
> ease of setup (one login for mail receive/send, contacts and calendars).
> 
> # Why is JMAP better than IMAP?
> 
> JMAP not a conversion of IMAP to JSON; it is a new protocol. It was
> designed to be make much more efficient use of network resources, be
> easier for developers to work with and hopefully make the best
> protocol for email an open standard once more. It's based on years of
> experience and real-world experimentation at FastMail, and talking to
> other major MUA/MTA developers to make sure we understand the common
> needs of the industry.
> 
> Some important attributes that help achieve these goals:
> 
> * The protocol is stateless. It doesn't need a persistent connection,
>  which is better for mobile use which may have intermittent network
>  access and wants to conserve battery life by turning the radio off
>  whenever possible.
> 
> * Ids are immutable and not intended to be user visible. So folder
>  naming becomes less messy - more like NFS or filesystems with inodes
>  rather than a name-based hierarchy, and renaming is easy to detect and
>  cheap to sync.
> 
> * It has a flexible set of commands, which can be batched in arbitrary
>  ways.  You can batch or pipeline single JMAP operations over a stream
>  protocol easily enough if you wanted to, but we're mostly envisaging
>  it being used for stateless batch operations to make disconnection
>  less painful.
> 
> With IMAP you can set two messages to both have the same flag (. STORE
> 1,2 +FLAGS (aflag)) but you can't store two different flags to two
> different messages in the same action. JMAP allows multiple create,
> update and destroy actions on different messages in a single setMessages
> command. Pipelining also has the problem that if the connection drops at
> just the wrong moment you can wind up applying the first change but not
> the second.
> 
> You can use backreferences to other objects created in the same batch -
> allowing you to, for example, create a folder tree by referencing
> previous parents created in the same request.
> 
> * Clients can efficiently fetch updates from their current state a-la
>  QRESYNC. This can be implemented effectively using the MODSEQ data
>  already in modern IMAP servers, or by using a transaction log data
>  structure. The server can always indicate to the client if it
>  cannot calculate updates from a particular client state (e.g.
>  because it is too old).
> 
> * Flood control. The client can always restrict how much data the server
>  should send. For example, a command might return a "tooManyUpdates"
>  error if it exceeds the client's limit, rather than returning a
>  million "* 1 EXPUNGED" lines as can happen in IMAP. Sometimes it's
>  just more efficient to throw away cached data and refetch, especially
>  if you're a mobile/webmail interface with only a partial cache of the
>  server's data.
> 
> * It doesn't require a custom parser. I've got a longer explanation to
>  the HTTPS/JSON question lower down, but having an encoding format that
>  is well understood and has widespread support among all programming
>  languages makes it far easier for developers to get started,
>  especially if they don't want to build a whole MUA but just integrate
>  something with email.
> 
> * The data model is backwards compatible with both IMAP folders and
> gmail-
>  style labels. Servers that implement JMAP are likely to want to
>  support IMAP as well for the foreseeable future, so it's important to
>  be able to have data structures that support both. Messages are
>  similarly immutable other than flags/mailboxes.
> 
> * Email can be sent using the same protocol, reducing confusing failure
>  modes for users (again I talk more about this below). We also have
>  pretty-
>  much complete specs for calendaring and contacts via JMAP, but we're
>  not pushing for them to be standard yet because the object format is
>  still undergoing a lot of work in the CalConnect group. We think a
>  single consistent protocol for all of these has a lot of advantages
>  though, and we hope to get there in the future.
> 
> # Why use HTTPS/JSON?
> 
> The short answer is it's good enough, widely understood and it's by far
> the easiest thing for developers to adopt. There's support in basically
> all OSes and programming languages. It's easy to read and debug.
> 
> HTTP doesn't tend to run into firewall issues, and is so commonly used
> it has integrations which can help with optimisation (for example, iOS
> has built-in support for optimising radio usage by batching HTTP calls
> from different apps where possible, which their mail team have told us
> they would like to be able to use). This isn't an innate advantage of
> HTTP, but rather an advantage of its ubiquity.
> 
> With GZIP, JSON data is reasonably compact and fast enough to
> serialise/parse. However, the encoding/transport part of JMAP is not
> core to its operation, so future specifications could easily add
> alternatives (e.g. WebSocket (https://tools.ietf.org/html/rfc6455)
> instead of HTTPS, CBOR (http://tools.ietf.org/html/rfc7049) instead of
> JSON). For the initial version though, HTTPS+JSON makes the most sense.
> 
> # Binary data
> 
> Binary data is not transported in the JSON (and indeed, as has been
> pointed out, can't be without base64 encoding or similar, which is
> inefficient). Instead, attachments are referenced by a blobId, and
> uploaded/downloaded separately via HTTPS. Clients can reference the
> blobId elsewhere to, for example, attach the same file to a new message
> without having to download and reupload it again, a big win on slower
> internet connections.
> 
> This also means that regularly saving drafts (a common client behaviour)
> does not mean sending the same full multi-megabyte attachments over the
> network every 60s or so.
> 
> As it's out-of-band with the API calls, uploading/downloading files can
> easily be parallelised and doesn't block other API operations.
> 
> # Representation of email
> 
> JMAP defines a JSON structure that represents in a consistent and
> structured way all the information that the vast majority of
> clients need from an RFC5322 message. The server deals with the
> complexities of MIME, encoding issues, parsing headers etc. The
> intention is that the server will still operate with RFC5322
> messages for storage and certainly transmission; the JSON
> representation is not intended to replace RFC5322, just relieve
> client authors from having to deal with it.
> 
> Clients that want to or need to (for example those doing PGP in the
> client) can still fetch the RFC5322 if needed. The message is
> represented by a blobId, and the raw bytes can be fetched using the same
> binary download mechanism as mentioned above.
> 
> # Message submission
> 
> Message submission is via means of an "outbox" folder. Messages are
> moved there to send. This was chosen over a separate "send" command for
> a few reasons. Firstly it's most consist with the rest of the API,
> making it easier for clients to implement offline support
> (synchronisation is the same as other changes you might make to
> messages). Secondly, clients/servers can support delayed send
> (particularly useful for "undo send"), by simply setting the date on the
> message in the future and only sending from the outbox when this date is
> reached. Until then the message is just sitting in the outbox like any
> other mailbox, allowing client to list and revoke without needing custom
> API commands.
> 
> Error handling is flexible enough to return a full range of errors when
> you try to move a message to this folder, just as you would to a
> separate "send" method.
> 
> Clients can use the same JSON structure for sending messages as they get
> from the server for received messages, allowing the server to deal with
> MIME encoding. This allows clients to be much simpler and easier to
> write. (They can also upload a raw RFC5322 message if they want instead
> of course.)
> 
> Having the same protocol for message sync and submission is a huge win
> for usability; we see a lot of support tickets where users can receive
> but not send, or vice versa, because one of these is misconfigured. This
> is always very confusing for regular users.
> 
> # Push mechanism
> 
> Immediate updates is an important feature to many users. IMAP IDLE has
> two big problems: firstly it only notifies of changes in one folder, so
> doesn't inform you of all changes unless you open a connection for every
> folder, and secondly it requires a persistent network connection which
> is bad for mobile (and not even allowed on iOS).
> 
> JMAP defines two push mechanisms to support the two common use cases. In
> both cases the data transferred is simply an edge trigger: a new state
> string letting the client know something has changed within a particular
> datatype. The client then fetches the new data using the standard
> synchronisation methods.
> 
> For desktop clients and webmail, there's an event source interface
> (https://html.spec.whatwg.org/multipage/comms.html#the-eventsource-interface)
> . This requires a persistent HTTP connection.
> 
> For mobile, and web integrations, you can set a callback handler. This
> makes the mail store server do a callback to a server defined by the
> client when something changes; the client's server can then send out-of-
> band push events using the native push mechanism of the mobile client.
> JMAP itself doesn't require any particular mobile push technology.
> 
> # End-to-end encryption
> 
> A lot of the optimisations for efficient client-server sync require the
> server to be able to read the message. If everything were encrypted, the
> server would basically be a dumb blob store, unless you have some clever
> partial metadata search capability. This is particularly bad for mobile,
> where you only want to sync partial information. Users expect to be able
> to search their whole archive, so either you need all the data in the
> client, or the server needs to have access to the data.
> 
> JMAP is therefore not making any new measures to address end-to-end
> encryption. The best advice is probably to run your own "JMAP SERVER" on
> trusted hardware; otherwise you need to sync the entire multi-gigabyte
> mail spool to all your devices. JMAP is also simple enough that you
> could run the server on multiple machines with an underlying replication
> protocol over encrypted links and have that do your smarts.
> 
> # Draft proposals and implementations
> 
> As Arnt pointed out, "rough consensus and running code" are key here,
> and the current draft of the JMAP spec is being implemented in Cyrus
> IMAPd and Dovecot (the two largest open-source IMAP servers), as well as
> other open source projects like Apache James and Linagora. Roundcube
> have stated they plan to build their next-generation client on JMAP.
> 
> On the proprietary side, Atmail have a JMAP proxy and webmail client
> they are releasing to production very soon, and we at FastMail have a
> version of our client that talks JMAP too.
> 
> There is interest among other large mailbox providers/mail client
> authors, but they don't tend to be early adopters as much.
> 
> The current specification drafts can be found at:
> 
> https://datatracker.ietf.org/doc/draft-jenkins-jmap/ (The generic JMAP
> protocol)
> https://datatracker.ietf.org/doc/draft-jenkins-jmapmail/ (Mail over
> JMAP)
> 
> We have also developed an open-source JMAP proxy, which you can connect
> to an IMAP server to try out the protocol and see what happens over the
> wire. There's a hosted version at http://proxy.jmap.io/, or you can grab
> the code from https://github.com/jmapio/jmap-perl
> 
> Cheers,
> 
> Neil.
>