Re: Communciator 4.02 Imap EXPUNGE problem

[Mark Crispin <[e-mail]>]

On Tue, 26 Aug 1997 09:20:07 -0400 (Eastern Daylight Time), Barry Leiba,
Multimedia Messaging wrote:
> > But, it assumes that when the official, documented, mechanism fails, an
> > unofficial and undocumented alternative will work in a defined manner.
> There are often multiple, valid ways of doing things.  In the absence of a
> statement to the contrary, and because the IMAP spec give one a strong
> reason to believe that multiple clients can connect to the same mail store,
> this is a reasonable assumption -- that's all I'm trying to say.

I understand, and to some extent agree.

However, RFC 2060 doesn't quite state that as strongly as you suggest.
"Simultaneous access to the same mailbox" is referenced in passing only three
times, with such wording as "Server implementations that offer multiple
simultaneous access to the same mailbox".  This implies that there can be
implementations that do not offer multiple simultaneous access to the same
mailbox; and there is no definition about what happens if a client attempts it
on a server which does not offer it.

The use of multiple connections in this way assumes that a server can be
kludged into doing something that it does not support.  It assumes the
presence of a facility that may not be absent and has undocumented results if
that facility is absent.  It assumes (in the case of Communicator 4.02) that
there are no locks that prohibit EXPUNGE when multiple sessions have the same
mailbox selected.  That's a lot of assumptions to make!

The bottom line is that there is an official documented mechanism to have
multiple operations in progress on the mailbox, and that mechanism should be
used.  Kludges just get you into trouble.

> However, the facts are that servers don't support this now, that even when
> they do there's no way the client can know whether the server does or not,
> and that it's easier for both the client and the server to code it the
> other way.

The specification was carefully designed so that a client or server which
supports it can interoperate with one that does not.  There is no such
guarantee that a client that does the kludge can interoperate.

Interoperability is what makes a good implementation.  If an implementation
does not interoperate with other compliant implementations, then no matter how
fancy its GUI may be it will be a poor implementation.

> This is partly because of the untagged
> response thing... if all responses were tagged to indicate which command
> caused them, and untagged responses were reserved for asynchronous
> communications, it would be much easier to implement this on the client
> side.

This suggests to me that you do not understand the data model behind IMAP;
this is what lead to the IMAP3 debacle.  Clients are supposed to operate on a
single client-based cache of the mailbox, and go out to the server when the
client cache faults.  It doesn't matter what object triggered the fault.
Untagged data are server commands to update the client-based cache; don't
think of them as command responses.

Yes, I know that this effectively prevents having multiple SEARCHes in
progress at a time.  But suppose such a facility existed, and a SEARCH takes
30 seconds.  So, instead of doing two SEARCHes in sequence, getting the first
after 30 seconds and the second after 60 seconds, you do them simultaneously
and get them both after 60 seconds.  I suspect that it's not worth doing
anything about it until we have parallel processing IMAP search engines.

> Perhaps we should add a CAPABILITY token for this -- some token that the
> server returns in the CAPABILITY response that indicates its support for
> multiple concurrent commands in the same session.

My contention is that a client that wants to use this facility should always
use it, unconditionally.  This will work with either type of server.

> A *message* can disappear, but a sequence number can't.

Not true.  If you expunge 100 messages from a 150 message mailbox, then
sequence numbers 51:150 are history.

> Those tend to be the mail stores on Unix file systems.  That's the past,
> but not the future; more and more, mail servers are being deployed with
> database back-ends, and they can deal with this situation well and
> efficiently.

It isn't just UNIX file systems; and not everyone wants to use a database back
end for their mail.

> >  1) multiple server processes can select the same mailbox, and it is not
> >     detrimental (e.g. kill other server behavior) to attempt to do so.
> >  2) EXPUNGE is permitted while another process has the mailbox selected.
> >  3) EXPUNGE does not block other processes which have the mailbox
> >     selected.
> >  4) the message being fetched can't be expunged in mid-FETCH
> >  5) EXPUNGE takes enough time that a performance benefit would be realized
> >     by doing it.
> > I know of no servers in which all of the above are true.
> I do.  More than one.  They use database back-ends.

Did you take note of (5)?

I'd be rather disappointed in a database back end that had a slow EXPUNGE!
That would lose 95% of the advantage of a database back end.

> Again, this is
> exactly why something needs to be said about it -- because someone
> implementing a client and testing against servers that use  more flexible
> back-ends might be unaware of the problems associated with using those
> kinds of mail stores.

I guess the problem I have with this is that it states the obvious!

How many times do we have to say "don't make unwarranted assumptions"?  How
many times do we have to say "don't assume that if, your client works with one
server, it is a correct implemtnation and you don't need to do any more
testing"?

We can name examples, such as this one, where there was a problem in the past.
But that isn't going to prevent new cases from popping up in the future;
humans are quite creative in the realm of "unwarranted assumptions".

It certainly does not belong in the specification, which is large and bloated
enough as it is.  Maybe a document of "case histories of how people screwed up
interoperability" might be useful.  But I think that the lesson to be learned
is not any of the specific case histories, but the general lesson of "assume
the worst out of the other side."

> Nevertheless, if you want to deprecate a client behaviour that clearly is
> following the protocol and that "on the surface, [...] seems reasonable",
> then you need to do it somewhere where the client implementers are going to
> find it.

There's nothing to deprecate!

It was never sanctioned, it requires making three unwarranted assumptions, and
it is clearly a kludge when there is a documented manner.

It is not the purpose of a specification to outlaw silly behavior.

> It's not reasonable to assume that they will test their clients
> against your server and will happen to use one of the two (of the nearly a
> dozen) mail stores that show the problem.

I *strongly* disagree with this statement.

We're talking about what is, and is likely to remain for a long time, the most
common IMAP server in its default configuration.