Re: [imapext] [ietf-smtp] Fwd: Request to form a new WG: JMAP
Neil Jenkins <neilj@fastmail.com> Fri, 11 November 2016 03:57 UTC
Return-Path: <neilj@fastmail.com>
X-Original-To: imapext@ietfa.amsl.com
Delivered-To: imapext@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A5AD11298C7 for <imapext@ietfa.amsl.com>; Thu, 10 Nov 2016 19:57:49 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.72
X-Spam-Level:
X-Spam-Status: No, score=-2.72 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=fastmail.com header.b=h4PzMdq5; dkim=pass (1024-bit key) header.d=messagingengine.com header.b=UZt5Fq+I
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xFMaheV89clj for <imapext@ietfa.amsl.com>; Thu, 10 Nov 2016 19:57:46 -0800 (PST)
Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9DA971297C1 for <imapext@ietf.org>; Thu, 10 Nov 2016 19:57:46 -0800 (PST)
Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id B3FF82065C for <imapext@ietf.org>; Thu, 10 Nov 2016 22:57:45 -0500 (EST)
Received: from betaweb1 ([10.202.2.10]) by compute7.internal (MEProxy); Thu, 10 Nov 2016 22:57:45 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=fastmail.com; h= content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=mesmtp; bh=1R0qHXqtL5vwSMqijLDyJlTrxz k=; b=h4PzMdq59go0qGQOeV2S9nNEzMW/gUxXsxMbl8mlyUT1c4NsbErtTABEY2 fnyPGzkhSdBirMLQonRYb5yTBvnzJuLbSAr3f7t5OA5WCj690P6gF7RVJOindugg 5/qVfb+dKCibFUUjkvIxcXRSIuC64gkUwQQmXAQCRE3AuaRJI=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=smtpout; bh=1R 0qHXqtL5vwSMqijLDyJlTrxzk=; b=UZt5Fq+Ik3iqg5zpO5mxS6Gua9fnylxW1w fGJQxOgxqkCKF4ej0ksPvq1UdLtwLiEmOfA4BNtw0QcAacmogUfBAxO9sQFlnKOz j0+tf2+Vx6DyFG2jKOy0O5Es1YVtntD/MVkbgx0rwieR9mSiiAPfJoCC9mrLD1Hs 2QSzXpuw4=
X-ME-Sender: <xms:uUElWP4EXhRidI7hUfUeI-bLWe4IgEVD0rsyt_FU6p_8O8wc294f8g>
Received: by mailuser.nyi.internal (Postfix, from userid 99) id 70E2CE2889; Thu, 10 Nov 2016 22:57:45 -0500 (EST)
Message-Id: <1478836665.322873.784300457.3FB705B7@webmail.messagingengine.com>
From: Neil Jenkins <neilj@fastmail.com>
To: imapext@ietf.org
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"
X-Mailer: MessagingEngine.com Webmail Interface - ajax-d4793dc0
In-Reply-To: <E99C5826138657241662E74E@JcK-HP8200>
References: <1478539079.1706686.780110457.75B1F9CF@webmail.messagingengine.com> <a786d82d-7134-c7bc-24ef-5dfb56e7bbac@isode.com> <01Q7166TP70G011H9Q@mauve.mrochek.com> <b85870ed-86e0-0f97-fece-476399124e81@isode.com> <01Q74JCZZYFG00Z4TS@mauve.mrochek.com> <E99C5826138657241662E74E@JcK-HP8200>
Date: Fri, 11 Nov 2016 14:57:45 +1100
Archived-At: <https://mailarchive.ietf.org/arch/msg/imapext/wFxgIvlfweBFlgcDDZf9la6j8lg>
Subject: Re: [imapext] [ietf-smtp] Fwd: Request to form a new WG: JMAP
X-BeenThere: imapext@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Discussion of IMAP extensions <imapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/imapext>, <mailto:imapext-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/imapext/>
List-Post: <mailto:imapext@ietf.org>
List-Help: <mailto:imapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/imapext>, <mailto:imapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 11 Nov 2016 03:57:50 -0000
Hi all, Rather than reply to individual emails, I'm going to try to put across answers to the queries and concerns related so far in this thread. I'm the author of the draft specification we're putting forward with this charter; happy to answer any further questions. # What is JMAP replacing? JMAP is intended to be a new standard for email clients to connect to mail stores. It therefore replaces IMAP + SMTP submission. It is also designed to be more generic such that it can be extended with contacts, calendars in the future (replacing CardDAV/CalDAV). It does not replace MTA-to-MTA SMTP transmission. # Why is this needed? It's too hard to write a good MUA with current standards, which has led to a stagnation in good email clients and a proliferation of proprietary protocols. In addition, IMAP is really not suited to a constrained network environment, as often found on mobile networks even in developed countries. The chatty nature of the IMAP protocol is ill-suited to high latency connections. Things like a folder rename on the server can mean that all the downloaded and cached email for that folder is invalidated (because you can't tell for sure over the protocol that they're the exact same messages), wasting huge amounts of bandwidth. Stateful connections make it harder to deal with intermittent network outages. We're attempting to standardise a new protocol because lots of people are writing proprietary alternatives to deal with the same deficiencies with the current standards. Some of these deficiencies could be fixed by adding things to IMAP (for example persistent IDs on folders and messages for better caching when another client copies, moves or renames), but others are structural like the need to calculate a MSGNO <=> UID mapping even if the client doesn't want it, or the need to find endpoints and authenticate separately to multiple different protocols to do related tasks. Because it's so hard to write a good email client that works with any IMAP provider, these days many new clients are just for Gmail (with possibly support for a few other big mailbox providers). It's easy to see why: IMAP is either the woefully inadequate original RFC3501, or it's a messy set of incomplete implementations of some of the extensions. Even CONDSTORE (without QRESYNC) is pushing it; you can't rely on it to be there in most cases. As a result of this, proprietary protocols have been popping up as alternatives to IMAP. Here are a few examples, the latter two being from whole companies that formed just to try to help people not have to deal with IMAP: Gmail: https://developers.google.com/gmail/api/v1/reference/ Outlook: https://msdn.microsoft.com/office/office365/APi/mail-rest-operations Nylas: https://nylas.com/cloud/docs Context.io: https://context.io/docs/lite In addition, we're seeing most new mobile email clients proxy everything via their own server rather than talking directly to the user's mail store, for much the same reasons. Examples include Mailbox (now defunct), Alto, Outlook and Newton. This is bad for security and privacy, and also bad for the client authors as they have to run server infrastructure in addition to just building their clients. Despite not only being proprietary but patented (and expensive!), ActiveSync has seen a big increase in adoption, and not just with Microsoft servers, due to its better support for mobile environments and ease of setup (one login for mail receive/send, contacts and calendars). # Why is JMAP better than IMAP? JMAP not a conversion of IMAP to JSON; it is a new protocol. It was designed to be make much more efficient use of network resources, be easier for developers to work with and hopefully make the best protocol for email an open standard once more. It's based on years of experience and real-world experimentation at FastMail, and talking to other major MUA/MTA developers to make sure we understand the common needs of the industry. Some important attributes that help achieve these goals: * The protocol is stateless. It doesn't need a persistent connection, which is better for mobile use which may have intermittent network access and wants to conserve battery life by turning the radio off whenever possible. * Ids are immutable and not intended to be user visible. So folder naming becomes less messy - more like NFS or filesystems with inodes rather than a name-based hierarchy, and renaming is easy to detect and cheap to sync. * It has a flexible set of commands, which can be batched in arbitrary ways. You can batch or pipeline single JMAP operations over a stream protocol easily enough if you wanted to, but we're mostly envisaging it being used for stateless batch operations to make disconnection less painful. With IMAP you can set two messages to both have the same flag (. STORE 1,2 +FLAGS (aflag)) but you can't store two different flags to two different messages in the same action. JMAP allows multiple create, update and destroy actions on different messages in a single setMessages command. Pipelining also has the problem that if the connection drops at just the wrong moment you can wind up applying the first change but not the second. You can use backreferences to other objects created in the same batch - allowing you to, for example, create a folder tree by referencing previous parents created in the same request. * Clients can efficiently fetch updates from their current state a-la QRESYNC. This can be implemented effectively using the MODSEQ data already in modern IMAP servers, or by using a transaction log data structure. The server can always indicate to the client if it cannot calculate updates from a particular client state (e.g. because it is too old). * Flood control. The client can always restrict how much data the server should send. For example, a command might return a "tooManyUpdates" error if it exceeds the client's limit, rather than returning a million "* 1 EXPUNGED" lines as can happen in IMAP. Sometimes it's just more efficient to throw away cached data and refetch, especially if you're a mobile/webmail interface with only a partial cache of the server's data. * It doesn't require a custom parser. I've got a longer explanation to the HTTPS/JSON question lower down, but having an encoding format that is well understood and has widespread support among all programming languages makes it far easier for developers to get started, especially if they don't want to build a whole MUA but just integrate something with email. * The data model is backwards compatible with both IMAP folders and gmail- style labels. Servers that implement JMAP are likely to want to support IMAP as well for the foreseeable future, so it's important to be able to have data structures that support both. Messages are similarly immutable other than flags/mailboxes. * Email can be sent using the same protocol, reducing confusing failure modes for users (again I talk more about this below). We also have pretty- much complete specs for calendaring and contacts via JMAP, but we're not pushing for them to be standard yet because the object format is still undergoing a lot of work in the CalConnect group. We think a single consistent protocol for all of these has a lot of advantages though, and we hope to get there in the future. # Why use HTTPS/JSON? The short answer is it's good enough, widely understood and it's by far the easiest thing for developers to adopt. There's support in basically all OSes and programming languages. It's easy to read and debug. HTTP doesn't tend to run into firewall issues, and is so commonly used it has integrations which can help with optimisation (for example, iOS has built-in support for optimising radio usage by batching HTTP calls from different apps where possible, which their mail team have told us they would like to be able to use). This isn't an innate advantage of HTTP, but rather an advantage of its ubiquity. With GZIP, JSON data is reasonably compact and fast enough to serialise/parse. However, the encoding/transport part of JMAP is not core to its operation, so future specifications could easily add alternatives (e.g. WebSocket (https://tools.ietf.org/html/rfc6455) instead of HTTPS, CBOR (http://tools.ietf.org/html/rfc7049) instead of JSON). For the initial version though, HTTPS+JSON makes the most sense. # Binary data Binary data is not transported in the JSON (and indeed, as has been pointed out, can't be without base64 encoding or similar, which is inefficient). Instead, attachments are referenced by a blobId, and uploaded/downloaded separately via HTTPS. Clients can reference the blobId elsewhere to, for example, attach the same file to a new message without having to download and reupload it again, a big win on slower internet connections. This also means that regularly saving drafts (a common client behaviour) does not mean sending the same full multi-megabyte attachments over the network every 60s or so. As it's out-of-band with the API calls, uploading/downloading files can easily be parallelised and doesn't block other API operations. # Representation of email JMAP defines a JSON structure that represents in a consistent and structured way all the information that the vast majority of clients need from an RFC5322 message. The server deals with the complexities of MIME, encoding issues, parsing headers etc. The intention is that the server will still operate with RFC5322 messages for storage and certainly transmission; the JSON representation is not intended to replace RFC5322, just relieve client authors from having to deal with it. Clients that want to or need to (for example those doing PGP in the client) can still fetch the RFC5322 if needed. The message is represented by a blobId, and the raw bytes can be fetched using the same binary download mechanism as mentioned above. # Message submission Message submission is via means of an "outbox" folder. Messages are moved there to send. This was chosen over a separate "send" command for a few reasons. Firstly it's most consist with the rest of the API, making it easier for clients to implement offline support (synchronisation is the same as other changes you might make to messages). Secondly, clients/servers can support delayed send (particularly useful for "undo send"), by simply setting the date on the message in the future and only sending from the outbox when this date is reached. Until then the message is just sitting in the outbox like any other mailbox, allowing client to list and revoke without needing custom API commands. Error handling is flexible enough to return a full range of errors when you try to move a message to this folder, just as you would to a separate "send" method. Clients can use the same JSON structure for sending messages as they get from the server for received messages, allowing the server to deal with MIME encoding. This allows clients to be much simpler and easier to write. (They can also upload a raw RFC5322 message if they want instead of course.) Having the same protocol for message sync and submission is a huge win for usability; we see a lot of support tickets where users can receive but not send, or vice versa, because one of these is misconfigured. This is always very confusing for regular users. # Push mechanism Immediate updates is an important feature to many users. IMAP IDLE has two big problems: firstly it only notifies of changes in one folder, so doesn't inform you of all changes unless you open a connection for every folder, and secondly it requires a persistent network connection which is bad for mobile (and not even allowed on iOS). JMAP defines two push mechanisms to support the two common use cases. In both cases the data transferred is simply an edge trigger: a new state string letting the client know something has changed within a particular datatype. The client then fetches the new data using the standard synchronisation methods. For desktop clients and webmail, there's an event source interface (https://html.spec.whatwg.org/multipage/comms.html#the-eventsource-interface) . This requires a persistent HTTP connection. For mobile, and web integrations, you can set a callback handler. This makes the mail store server do a callback to a server defined by the client when something changes; the client's server can then send out-of- band push events using the native push mechanism of the mobile client. JMAP itself doesn't require any particular mobile push technology. # End-to-end encryption A lot of the optimisations for efficient client-server sync require the server to be able to read the message. If everything were encrypted, the server would basically be a dumb blob store, unless you have some clever partial metadata search capability. This is particularly bad for mobile, where you only want to sync partial information. Users expect to be able to search their whole archive, so either you need all the data in the client, or the server needs to have access to the data. JMAP is therefore not making any new measures to address end-to-end encryption. The best advice is probably to run your own "JMAP SERVER" on trusted hardware; otherwise you need to sync the entire multi-gigabyte mail spool to all your devices. JMAP is also simple enough that you could run the server on multiple machines with an underlying replication protocol over encrypted links and have that do your smarts. # Draft proposals and implementations As Arnt pointed out, "rough consensus and running code" are key here, and the current draft of the JMAP spec is being implemented in Cyrus IMAPd and Dovecot (the two largest open-source IMAP servers), as well as other open source projects like Apache James and Linagora. Roundcube have stated they plan to build their next-generation client on JMAP. On the proprietary side, Atmail have a JMAP proxy and webmail client they are releasing to production very soon, and we at FastMail have a version of our client that talks JMAP too. There is interest among other large mailbox providers/mail client authors, but they don't tend to be early adopters as much. The current specification drafts can be found at: https://datatracker.ietf.org/doc/draft-jenkins-jmap/ (The generic JMAP protocol) https://datatracker.ietf.org/doc/draft-jenkins-jmapmail/ (Mail over JMAP) We have also developed an open-source JMAP proxy, which you can connect to an IMAP server to try out the protocol and see what happens over the wire. There's a hosted version at http://proxy.jmap.io/, or you can grab the code from https://github.com/jmapio/jmap-perl Cheers, Neil.
- [imapext] Fwd: Request to form a new WG: JMAP Alexey Melnikov
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … John C Klensin
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ned Freed
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … John C Klensin
- Re: [imapext] Fwd: Request to form a new WG: JMAP Arnt Gulbrandsen
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Adrien de Croy
- Re: [imapext] Request to form a new WG: JMAP Julien ÉLIE
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Doug Royer
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ned Freed
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … John C Klensin
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Arnt Gulbrandsen
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Alexey Melnikov
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Alexey Melnikov
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Doug Royer
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ned Freed
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … John C Klensin
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Neil Jenkins
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Tony Finch
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ned Freed
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ned Freed
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ted Lemon
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Doug Royer
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ted Lemon
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Bron Gondwana
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ned Freed
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Ted Lemon
- [imapext] offline mode, was Re: [ietf-smtp] Fwd: … Tony Finch
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Doug Royer
- Re: [imapext] [ietf-smtp] Fwd: Request to form a … Bron Gondwana
- [imapext] Further discussions of JMAP Alexey Melnikov
- [imapext] I moved to jmap@ietf.org Doug Royer
- Re: [imapext] [ietf-smtp] Request to form a new W… Jacob Palme
- Re: [imapext] offline mode, was Re: [ietf-smtp] F… Brandon Long