Re: [imap5] Designing a new replacement protocol for IMAP

[Adrien de Croy <[e-mail]>]

Hi Cyrus

I guess we are on different pages for what we want out of a mail 
protocol and where we want it to go.  Sorry if this mail rambles a bit.

Also taking into account Mark and others' comments.

Maybe we need to start with what we want to achieve... which actually 
Bron did do at the start.  All of us have various different histories 
when it comes to mail.

When I first implemented our IMAP server, I was firstly struck by the 
unusual syntax, parsing requirements, and then the missing bits - lack 
of completely defined folder management protocol support, and the highly 
problematic dual-indexing (no UID in expunge response!!!).   History 
obviously plays a huge part, but it's not 1998 any more.

Then I was hit with the quirks of the implementations (e.g. TB ignores 
unsolicited responses unless it issued an IDLE).  I've spent a lot of 
time evaluating IMAP clients, none of them are what I would call great 
(for my use).

In terms of features, it seems over half the feature set nowadays is 
optional protocol extensions, which basically means no-one can count on 
them being there.

So, I'm approaching this whole thing from a mail-user perspective, since 
I also am one.  I think usability could be greatly improved if the 
protocol supported it.

My issue with XMPP isn't with the protocol per-se, but just that if 
you're outside a firewall, someone needs to open another hole so you can 
access the XMPP server.  It introduces another point of failure for the 
system.  If we were to rely on XMPP for notifications, we would just be 
adding support tickets like "my mail client doesn't notify me when I get 
new mail".  I already get "I can retrieve mail but not send".  Also you 
then need to have hooks into it at both ends, and define some sort of 
addressing system so the XMPP server can know what the client wants it 
to provide notifications about, and hooks in the back end of the 
services so it can get them.  Lots of points of failure.

I don't know how many of you spend a lot of time on customer support 
desks.  I do.

So I think maybe we need to take a step back.  If we want people to 
implement something new, it has to have some compelling reasons.  I 
don't think just interop is that compelling since it's not completely 
awful at the moment, and I think that is not thinking big enough.  We 
could do so much more.  If we're going to do a new mail protocol, we 
need to think what we want it to be like, with the benefit of however 
many decades of experience.

Mark is right in that the problems that IMAP solves are often problems 
that will also need solutions in a new system.  I don't believe 
implementing a new protocol means doing all that work again.  I've 
refactored enough code to know the thing you keep is the knowledge, even 
if the syntax changes.  So, anyway for me an incremental change to IMAP 
isn't horribly interesting.  Sure we can fix some things, but the way 
the protocol is extended and requirement for backwards compatibility is 
always going to hamstring us.

For me something more interesting would be a protocol that allows a 
client to provides the best possible mail user experience. And doing 
that using a single connection.

The reason for a single connection (could maybe be relaxed to a single 
port) is simply to reduce deployment issues.  One username, one 
password, one port to open.  That's the minimum number of things to go 
wrong, and minimum number of things to support.

If a vendor knows that by moving to a new mail protocol their customer 
support issues can halve, that's very compelling.
If a vendor knows that by using a new protocol they can drastically 
improve their users' mail experience, that's very compelling.

For me, I'd like something that:

* is fast, even over low bandwidth or latent connections
* is simple to deploy for administrators and users
* gives me all the mail functions I could need, including calendar etc.
* gives me cool new functions, like telling me the mail address I just 
typed into the destination field is bad even before I finished typing 
the subject line.
* works well with multiple clients (e.g. home and office, and on the road)

And no, I don't believe for a second this will happen overnight.

Cheers

Adrien


On 14/02/2012 4:20 a.m., Cyrus Daboo wrote:
> Hi Adrien,
>
> --On February 13, 2012 10:56:49 AM +1300 Adrien de Croy 
> <[e-mail]> wrote:
>
>>> So, in my opinion, whilst push notifications should be a requirement
>>> for imap5, we should not define that protocol and instead push the
>>> IETF to provide such a protocol for general use.
>>>
>>
>> I don't think that's a workable approach.
>>
>> Getting such a protocol together, which enables notifications from any
>> other application protocol I think will take a very long time, if it can
>> even succeed.  It's hard enough getting consensus within one protocol
>> working group, let alone all of them working together.
>>
>> Also every different protocol has different notification requirements.
>> trying to cover all that in a single protocol I think would be 
>> difficult.
>
> I disagree because what I envisage for the generic notification 
> service is an OS-level api (supplied by OS or 3rd party libraries) 
> that implement the "internet push service protocol". What that means 
> is that client developers only have to implement a simple api to get 
> push notifications. What is more anyone implementing more than one 
> protocol in their client (e.g. IMAP, CalDAV and CardDAV) would only 
> have to implement that once, albeit with some minor differences in 
> regard to how to get protocol specific pieces for registering for 
> notifications.
>
> Your point about actually getting this done is valid. But 
> realistically, do you really think IMAP5 is going to deploy overnight? 
> Frankly, anyone who has a reasonably solid IMAP4 implementation today 
> is going to question the need to work on something new, particularly 
> if that something new brings nothing to the table (and simply fixing 
> interop problems will probably not be seen as something "new"). If you 
> can actually show that IMAP5 adds significant value by doing things 
> like helping centralize push notifications, simplifying submission 
> etc, then maybe, just maybe, those existing implementors might 
> actually consider throwing out their current investment in IMAP4 for 
> the new thing. But, IMHO, you are really going to have to up-sell 
> IMAP5 to get buy-in from the major email providers. Now that does not 
> mean it is not worth doing, but it does mean having to do more than 
> just fixing perceived or real deficiencies.
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com