Re: Communciator 4.02 Imap EXPUNGE problem

[Mark Crispin <[e-mail]>]

On Mon, 25 Aug 1997 15:51:20 -0400 (Eastern Daylight Time), Barry Leiba,
Multimedia Messaging wrote:
> I agree with you if we're talking about a client that spawns multiple
> processes to try to pump more work through the server (as many HTTP
> clients do).

Yes, I am.

In my experience, when I have seen clients spawn multiple server sessions, it
has been for one of the following reasons:
 1) The client needs to do something outside of the context of the current
    SELECT.  Examples of this include wanting to open multiple mailboxes
    simultaneously, wanting to open another mailbox but be able to keep the
    current selected mailbox if there is an error (in this case, you open a
    second session, and close the first session & switch over only if the open
    of the second session succeeds), or checking for new mail in another
    mailbox (the STATUS command largely eliminates this).

    You need to be reserved when doing this, but this is acceptable behavior
    and a good example of when you need multiple processes.

 2) The client thinks that it can pump more work through the server with
    multiple processes.  I think that we are in agreement that this is bad
    behavior; it is practiced by many clients of certain other protocols such
    as HTTP; and that it should be discouraged.

 3) The client thinks that it can get something else done during a time
    consuming operation (as you indicate in your next paragraph; I'll discuss
    this below).

> In this case, though, we're not talking about that - we're
> talking about the idea of a client's managing *two* (not "a bunch of")
> server processes: one to which it sends work that it thinks might be
> long-running and another to which it sends short-running work for which
> relatively prompt interactive response is desired.

IMAP defines a mechanism for doing multiple things to the same mailbox at the
same time; refer to RFC 2060 section 5.5.  I personally have never liked that
idea, and the fact that it is not widely deployed should suggest the
difficulty in going from ideal to reality.  But, that is the defined
mechanism.

To paraphrase your paragraph, "the defined IMAP mechanism isn't widely
deployed, so we can work around it by having two server processes on the same
mailbox, one in which we do the slow thing, and one in which we do fast
things."  On the surface, this seems reasonable.

But, it assumes that when the official, documented, mechanism fails, an
unofficial and undocumented alternative will work in a defined manner.

The two most likely "slow" operations are SEARCH and EXPUNGE, and the desire
is to do a FETCH during these.  Let's consider these.

SEARCH is an excellent example in which one may want to do a FETCH in the mean
time.  It is what section 5.5 envisions.  My personal belief is that client
implementors should lobby to get section 5.5 supported in servers with a non-
instantaneous SEARCH.

EXPUNGE is a different can of worms.  You have sequence numbers changing.  A
message can disappear out from under the client.  UID FETCH is much more
practical than FETCH in this case, but you still can have messages
disappearing.  Lots of clients have had real problems with this!

Many mail stores require an exclusive lock when doing an EXPUNGE (either
refusing an EXPUNGE when the share count is greater than 1, or blocking all
sharers until the EXPUNGE is complete).  Usually, the mail stores which
require the exclusive lock are also those in which EXPUNGE will take a
noticable amount of time; for example, servers which rewrite a flat file.

So, it is useful to do a FETCH in another session during an EXPUNGE (which
Communicator 4.02 is reported to do) only if all of the following are true:
 1) multiple server processes can select the same mailbox, and it is not
    detrimental (e.g. kill other server behavior) to attempt to do so.
 2) EXPUNGE is permitted while another process has the mailbox selected.
 3) EXPUNGE does not block other processes which have the mailbox selected.
 4) the message being fetched can't be expunged in mid-FETCH
 5) EXPUNGE takes enough time that a performance benefit would be realized
    by doing it.
I know of no servers in which all of the above are true.

> If it's a
> particularly bad idea in IMAP, owing to elements of server back-ends which
> may not be obvious to many implementers, then it's worth making note of it.

There are extant mail stores which do not (can not) permit multiple
simultaneous access to a mailbox.  This isn't something that a server
implementor can do anything about.

This restriction is enshrined in the POP protocol.

Now, unlike POP, IMAP permits multiple simultaneous access if the underlying
mail store allows it.  That "if" is important.  IMAP does not guarantee that
multiple simultaneous access (or folders other than INBOX) are available; it
merely makes them available to the client if they are available.

"Well, then, why not announce whether or not your support it as a capability?"

That's easier said than done.  My server supports multiple simultaneous
access.  It's a question if the mail store supports it or not.  My server can
access nearly a dozen different mail stores, only 2 of which do not support
multiple simultaneous access.  So, "support for multiple simultaneous access"
is on a per-mailbox basis.  To top it off, there is often no way to know
whether or not it is supported except by opening the mailbox.

It may be reasonable to add something at SELECT (and maybe also STATUS) time
to report whether or not that mailbox offers multiple simultaneous access.

But, I consider it to be much more important to press ahead of deploying 5.5
if client implementors really want this capability.  That is what the protocol
defines as what should be done, and that is what people should be
implementing.