Re: [imapext] multisort proposals (was Re: draft-ietf-appsawg-multimailbox-search)

[Bron Gondwana <[e-mail]>]

On Tue, Mar 11, 2014, at 07:43 AM, Chris Newman wrote:
> It sounds like IMAP cross-folder sort is a big non-interoperable mess
> right now due to lack of standardization.
>
> Have any of you considered writing a specification of your multisort
> protocol? Or maybe arrange a phone call between proponents of the
> "xconvmultisort" proposal, the "X-MAILBOX, X-REAL-UID" and the "\all"
> proposals to see if consensus on a standard approach is possible?

Apologies for the late reply on this - I've been busy dealing with a
nasty server partial-failure which led to corruption, and the associated
cleanup and recovery.  The "joy" of devops.

> Standards have several benefits for the vendors who participate:
>
> 1. If you have an early implementation, that's a key marketing/selling
>    point.
>
> 2. It gives your customers more choice of the other side when they
>    choose your product. (more client options if you're a server
>    vendor, or vice-versa if you're a client vendor).
>
> 3. It improves your reputation. It shows you care about the future of
>    your customers and the future of the Internet.
>
> 4. It raises the innovation baseline. Once a standard is written and
>    successful, it becomes "common practice" over time. Then there's no
>    more need to proselytize that standard, new innovation can start
>    with that standard as a baseline.
>
> 5. Once a standard is approved, you're much less likely to have to
>    waste time and effort implementing two or more ways of doing the
>    same thing to interoperate with different implementations.
>
> 6. It's one way a smaller vendor can get a leg-up on a bigger vendor
>    who ignores standards.
>
> While private IMAP extensions are fine for experiments, they're an
> interoperability problem when more than one for the same purpose show
> up. Why not take the time to write and progress a protocol
> specification so you and your customers reap the benefits and the
> community's innovation baseline is raised?

We have indeed.  Greg Banks (who doesn't work at FastMail any more, but
wrote the initial conversations support) did some initial work towards a
standards proposal for our conversations.  It was paused mainly because
we wanted to iron out the issues which would become obvious with real-
world usage first.

We have also done some work towards a more general mail access protocol
over HTTP, which is shared at http://jmap.io/

My general guiding principle is that the client should be describing
what it wants as clearly as possible, and the language of the protocol
needs to have the facility for it to do so.

The secondary principle is that the server should never do work that the
client has not explicitly requested.

This means that MSGNO by default is out, because that's an MVCC snapshot
of mailbox data.  It's an ordering that existed at a point in time, and
needs to be maintained by the server for an unbounded length of time.
The client can issue commands every 29 minutes which are chosen to not
allow EXPUNGE notifications - and the server can't forget those messages
until the mailbox is unselected.

Also, UNSEEN and RECENT have costs to calculate which aren't required by
many clients.

One thing that really frustrates me is optimisation hacks encoded in the
standard and guidance.  Things like "do a listing with %, then recurse
into each sub mailbox".  It might be more efficient when you're talking
to a server which is storing the folders in a filesystem and doing the
moral equivalent of 'ls' on each directory.

It's significantly less efficient with Cyrus, where the mailboxes database
is quite efficient to iterate in mailbox order (including sub-folders).
You're basically inspecting the sub-folders anyway - at least until you
can generate the \HasChildren or otherwise.

Generally, I would like the client to be able to say "I want the full
folder tree thanks, but limit it to the first 1000 items".

So my ideal protocol for a X-MULTISEARCH would smell a bit more like SQL
I suspect.  The offset / limit or anchor / limit syntaxes are both very
useful ways of describing exactly what you want to the server.

------

It would also be really great to merge the X-GM-MSGID and similar tricks
that various sites are using to uniquely identify messages across copy/move
operations that were performed by a DIFFERENT client.

APPENDUID is all very nice if you're using the client which did the COPY,
but it falls down if a different client did the copy - you wind up having
to fetch the entire message again if you don't trust the BODYSTRUCTURE and
ENVELOPE to uniquely identify the message enough - and most clients don't
due to the fact that Message-Id isn't 100% reliable in this broken age :(

I think those are the two areas.  I've said before that I'm willing to put
the time in to do this - and you're right - I really should stop talking
and start producing!

------

One final thing that we have found insanely useful in Cyrus replication:
checksums all over the place.  This is not so much useful to a traditional
"partial view" client, but many clients seem to synchronise the entire
contents of a mailbox, and a CRC would have value there.

The issue that's kept me from having time to reply for the past week is
documented here:

http://blog.fastmail.fm/2014/03/13/cleaning-up-from-an-imap-server-failure/

Having the sha1 of the spool files in our logs enabled data recovery in a
way that would have otherwise been impossible on an untrusted filesystem.
We could be sure that the emails we recovered were not corrupted.

Having a strong digest available for every message on the server would also
allow clients to consistency-check their local cache without resyncing all
the data.  I used to think that being able to detect mailbox renames that
you didn't do with this client was important (right now if another client
renames a folder, your client needs to fetch all the messages again for
the same reason as message copies) - but I think re-fetching just the flags/
annotations/digest isn't too expensive if you can re-use the cached message
contents.

Bron.


-- 
  Bron Gondwana
  [e-mail]