Re: [Jmap] message copying and moving

[Neil Jenkins <[e-mail]>]

On Wed, 2 Aug 2017, at 12:38 AM, Josef 'Jeff' Sipek wrote:
> I like the idea of the server presenting a property of some sort to
> indicate how many mailboxIds can be in the list for a message.  Even
> servers that support label-like mailboxes likely have some upper
> bound.  (Should null or absence mean 1?  unlimited?)
I would say null == "unlimited" (which of course is limited by the
number of mailboxes you have in the account). I've created
https://github.com/jmapio/jmap/issues/135 regarding this issue.
> As far as making it a per-account capability is concerned, that
> seemsincompatible with common Dovecot configurations where multiple
> storages are "grafted" into a single namespace.
You can split the storage of data within a JMAP account, but you'll
probably need a single index to fulfil the requirements for an account,
such as moving a message atomically between mailboxes, having threads
span messages across all mailboxes within the account, a unique id space
etc. Looking at the Dovecot docs, it seems your dbox format already
supports something like this (Alternate storage[1]) for storing the
actual data referenced by an index at different mount points. (We do
something very similar with Cyrus at FastMail to keep newer email on SSD
and older archives on spinning rust.) This is invisible to clients.
If you can't guarantee the properties required for an account for your
different mailboxes, then they have to be exposed as different
accounts. Methods that operate between accounts (e.g. copyMessages)
don't have the same constraints. A client can still choose to show them
together of course.
> To use the above two-storage setup as an example, suppose I want to
> copy a mail from INBOX to spam/misc.> 
> With IMAP, it'd be:
> 
> A SELECT INBOX
> B COPY 123 spam/misc
> 
> The server notices that the source of the message (INBOX) and the
> destination (spam/misc) are in different storages, and does a dumb
> copy instead of trying hardlink tricks.  The client doesn't care,
> because it knows that the new copy of the mail will have a new UID
> (one that's scoped to spam/misc).
So this is a clear case where they need to be different accounts, and so
the client can only use copyMessages to move from one to the other, not
setMessages. The semantics are then very similar to IMAP.
> Is there ever a time when someone may want to actually create a
> duplicate (IMAP COPY-style) without this sharing?  This automatic
> collapsing of messages could lead to some annoyed users that want two
> copies but with different keywords.
While I don't doubt there are some people that might want this, based on
the feedback we see, the vast majority of users would be happy to only
ever have a single shared copy that can exist in multiple locations.
With JMAP as currently defined, the server is free to support either (as
mentioned, we've decided with Cyrus to not allow this).
>> For example, the Cyrus implementation of JMAP
>> uses a secure hash of the message content to generate the
>> message id, so>> you cannot have two identical messages within the same user with
>> different>> message ids.
> 
> Out of curiosity, when you get a COPY or setMessages with a new
> mailboxId added, do you duplicate the mail itself (via a copy or a
> hardlink)?
Yes, because this was the easiest coming from the IMAP world where
each mailbox was completely separate, and there was no concept of a
"user". If I were to build a new server from scratch (and we have
thought about this a lot…) I wouldn't do this though; I'd just have a
blob store of immutable objects and a per-account index, and would
just update the index.
So to get from IMAP to JMAP with Cyrus we basically did:
 * Use secure hash of message content as message id.
 * JMAP mailboxes = set of mailboxes in which that message id exists.
 * When applying keywords with JMAP, we apply to every copy.
 * When reading keywords with JMAP, you get the union of keywords on
   all copies.
>> Looking at the spec, this needs to be clarified. However I believe we>> definitely need to make it so that (2)/(3) MAY return an existing
>> message>> id if the content is the same as an existing message (or
>> alternatively an>> error; either works for me).
> 
> I'm ok with MAY return existing messageId.
> …
>> * Allow servers to return an error instead of copy/import if the
>>   message already exists, rather than creating a duplicate with
>>   separate metadata. (But if it does succeed, then it MUST have
>>   separate metadata, be a newmessage id).> Sounds good.

OK, so as you noted we shouldn't *require* the server to anything
special  because that would force it to compare content hash and there's
no real need for this. So I think the options are:
 * *May* reject (error includes existing id) if content already exists.
   But if succeeds guaranteed to be separate copy with separate
   id/keywords/mailboxes.
 * *May* return  existing id if content already exists (and applies
   keywords/mailboxes like setMailboxes to the existing copy).
We need to pick one! I've created
https://github.com/jmapio/jmap/issues/134. I think probably the first
option, but don't feel too strongly either way.
> IMAP makes very few demands on what features the storage must provide.
> This no doubt helped its adoption since system administrators did not
> have to migrate all the users' mailboxes before exposing them via
> IMAP.  They just needed to install an imap server daemon, and point it
> at the maildirs. I think this is a very valuable property.> 
> Do we want jmap to also have this property?  Or do we want to relax it
> to some degree?
I think this has to be relaxed somewhat. I presume you could reasonably
easily migrate servers from mbox to mdbox right? I would have thought
sysops wouldn't care too much if you required them to migrate, as long
as the migration was easy and handled (mostly) automatically on upgrade.
Cheers,
Neil.

Links:

  1. https://wiki2.dovecot.org/MailboxFormat/dbox#Alternate_storage