Re: [imap5] Designing a new replacement protocol for IMAP

[Bron Gondwana <[e-mail]>]

On Tue, Feb 14, 2012 at 09:26:58AM +1300, Adrien de Croy wrote:
> Maybe we need to start with what we want to achieve... which
> actually Bron did do at the start.  All of us have various different
> histories when it comes to mail.

Oh yeah, so I did... it feels like a long time ago already!

> When I first implemented our IMAP server, I was firstly struck by
> the unusual syntax, parsing requirements, and then the missing bits
> - lack of completely defined folder management protocol support, and
> the highly problematic dual-indexing (no UID in expunge
> response!!!).   History obviously plays a huge part, but it's not
> 1998 any more.

Folder management is amazingly complex now that all the LIST-EXT
stuff is in there, and interactions with "subscriptions" - complete
with the whole "subscription has to survive when folder is deleted"
funkyness.  It's amazing how the best intentions have created a mess
there.

> Then I was hit with the quirks of the implementations (e.g. TB
> ignores unsolicited responses unless it issued an IDLE).  I've spent
> a lot of time evaluating IMAP clients, none of them are what I would
> call great (for my use).

Mark will agree there I think.  Me too.  I use offlineimap and mutt
with Maildirs still.  It's not awesome, but at least I can edit my
email in vim.

> In terms of features, it seems over half the feature set nowadays is
> optional protocol extensions, which basically means no-one can count
> on them being there.

I covered that in detail.  Optional is useless for a client writer,
because they have to support both options, and that's twice as much
code to write and debug and keep in sync.  So they don't bother
unless the benefits are huge - they just write to the lowest common
denominator.

> [ extra protocol is adds too many support headaches ]
> 
> So I think maybe we need to take a step back.  If we want people to
> implement something new, it has to have some compelling reasons.  I
> don't think just interop is that compelling since it's not
> completely awful at the moment, and I think that is not thinking big
> enough.  We could do so much more.  If we're going to do a new mail
> protocol, we need to think what we want it to be like, with the
> benefit of however many decades of experience.

Absolutely agree.  And my experience alone isn't enough, which is
why I don't want to go hide in a dark room and make one up out of
whole cloth.  I may have tried that a few years ago, but talking
to others at Fosdem in particular showed that I don't have all
the right answers.  I need to listen to other people who actually
believe in making this protocol too.  I'm willing to do a lot of
the hard labour in getting the implementations and TESTS done.
The tests are even more important than the code, because that's
how you keep implementations compatible.

I think that's something we have learned in the past 20 years.
Wordy specs don't do as much for compatibility as automated
tests which can be run against an implementation and show
immediately if it's doing the right thing or not.  Saying
"you fail test XYZ, and here's how you can test against it
until you have fixed your code" is much more effective than
saying "go read all the documents again you moron, and make
sure you understand them properly before you try to implement
anything"

> Mark is right in that the problems that IMAP solves are often
> problems that will also need solutions in a new system.  I don't
> believe implementing a new protocol means doing all that work again.
> I've refactored enough code to know the thing you keep is the
> knowledge, even if the syntax changes.  So, anyway for me an
> incremental change to IMAP isn't horribly interesting.  Sure we can
> fix some things, but the way the protocol is extended and
> requirement for backwards compatibility is always going to hamstring
> us.

And the first part of "not doing all the work again" is looking
at what we have now.  Every RFC and draft is the result of
somebody feeling enough pain from what's there now to sit down
and write up how they think their pain can be solved, and
shepherd it through the standards process.  That's a lot of
effort.  Those are important problems, and valuable thought
has gone into them.  Maybe some of them will be solved as a
side effect of discarding backwards compatibility at the
protocol level.  I sure hope so, because otherwise it's not
worth it.

> For me something more interesting would be a protocol that allows a
> client to provides the best possible mail user experience. And doing
> that using a single connection.

God yes.  That's what it's all about.

> The reason for a single connection (could maybe be relaxed to a
> single port) is simply to reduce deployment issues.  One username,
> one password, one port to open.  That's the minimum number of things
> to go wrong, and minimum number of things to support.

I think it has to be a single port.  Otherwise you wind up
having to multiplex EVERYTHING.  Ouch.  But one username,
one password, potentially a "sessionid" from the initial
login which can be used by the further connections.  I
think SCTP is not well enough supported to use, unfortunately,
because it would be a great way to deal with the multiple
connections issue otherwise.  There's a lot of potentially
great technology out there which just isn't right for the
problem.  We have to be pragmatic as well - it has to be
stupidly simple for administrators and users to deal with.

> If a vendor knows that by moving to a new mail protocol their
> customer support issues can halve, that's very compelling.
> If a vendor knows that by using a new protocol they can drastically
> improve their users' mail experience, that's very compelling.

Yes.

> For me, I'd like something that:
> 
> * is fast, even over low bandwidth or latent connections
> * is simple to deploy for administrators and users

These two are non-negotiable for me.

> * gives me all the mail functions I could need, including calendar etc.

This one is slightly more negotiable.  I think we could easily
over-reach in this.  I'd like to do a "version 1" which doesn't
go quite so far beyond what's there now.  That said, there's
the Kolab work to create a "Calendar Mailbox" which is just a
bunch of messages with a specified attachment type, one message
per calendar entry.  That plus a "\Calendar special use" or
similar could get you a working calendar system without needing
too much complexity.

> * gives me cool new functions, like telling me the mail address I
> just typed into the destination field is bad even before I finished
> typing the subject line.

That one's tricky.  I'd almost outsource that to a webservice
rather than being "in-protocol".  This being offline temporarily
is annoying, but not fatal to the user experience, and it's
cachable and doesn't necessarily need authentication.

> * works well with multiple clients (e.g. home and office, and on the road)

That's the one thing IMAP really has going for it already, but
we can get a bit better.

> And no, I don't believe for a second this will happen overnight.

Nup.

Bron.