Re: Communciator 4.02 Imap EXPUNGE problem

[Mark Crispin <[e-mail]>]

On Tue, 26 Aug 1997 13:54:37 -0400 (Eastern Daylight Time), Barry Leiba,
Multimedia Messaging wrote:
> I think, though, that most implementers *think of* IMAP
> as a protocol that allows simultaneous access to the same mailbox, and that
> it would be useful to state explicitly that some servers, under some
> conditions, do not, and that a client should not assume such capability.

I wonder how many other statements have to be made of the form "yes, the
protocol allows a server to export the concept but that doesn't necessarily
mean that it's there."

> First, the spec
> specifically refers to some things as "required responses".

Would it help to know that I didn't want to put that "required response" stuff
in the spec?  I was made to do it over my objections, because some people
thought it would make it "easier to understand" that way.  I think that it
obfuscated the model, and further perpetuated the myth that untagged data is
part of a response to a command.

Commands have only three responses: OK, NO, and BAD.  Untagged data are server
commands to update client state.  You are supposed think in terms of getting
your data from the client state, instead of thinking of from the server.

> Third, the lack of tags on the responses
> makes other things (like the indication that a message has gone away) more
> difficult than it should be.

I don't understand this claim.  "Message has gone away" is a very
straightforward event.  If it is difficult for a client to handle, then the
client has something wrong with its structure.

> Fair enough in the case we're talking about, but there are other cases when
> a client will want to know.  For instance, maybe I want to do a "FETCH
> 1:50 ALL" to populate the screen, wait for the response, and then "FETCH
> 51:* ALL" to let the rest of the data come through in the background.  But
> I still want to be able to get a "FETCH 5 BODY[1]" through if the user
> wants to read an item.

OK, this is a reasonable question.

The key to understanding how to handle this is the recognition that there are
four factors involved in FETCH real time consumption:
 1) server overhead (e.g. RFC822 or MIME processing)
 2) network bandwidth
 3) round-trip times
 4) client overhead

If you meter this, what you'll quickly discover is that (3) swamps all the
others with single FETCH, (1) may become signficant with a very large
aggregate FETCH in some servers, and (2) and (4) are relatively constant.
Actually, you don't even need to meter it; you can come to the same
conclusions via gedanken exercises.

The next thing that you discover is that (3) diminishes quickly with even
modest aggregation, whereas the others are linear.  You also have to consider
that even if you did a simultaneous "FETCH 5 BODY[1]" to a server that
supported it, there is no guarantee that the server would necessarily get it
done any faster.

So, here's how you handle it:

Pick some number, typically at least 20 and no more than 100, as an
aggregation size.  This is a value where the sum of (1), (2), and (4) are
relatively modest, but enough to reduce (3).  Let's say it's 50, which reduces
(3) by a factor of 50.

So, you do "FETCH 1:50 ALL", then in the background do "FETCH 51:100 ALL",
"FETCH 101:150 ALL", "FETCH 151:200 ALL", etc. in succession.  After each
step, you get an opportunity to slip in a "FETCH 5 BODY[1]" if needed.

"But," you say, "why should I need to do this on a server that supports
parallelism?"  The reason is that even if the server supports parallelism,
there is no guarantee that the server will return the results predictably.  It
is an unwarranted assumption that the server will recognize that the "FETCH 5
BODY[1]" is more important than a "FETCH 50:* ALL"!  In fact, one can come up
with scenarios in which it may be the other way around (maybe the ALL fetch is
more urgent to finish quickly than that 50GB video clip...)!!!!

If you use the aggregation size technique which I have outlined, you will have
predictable client behavior with all servers.

I've looked at several possible scenarios in which parallel FETCH might be
useful.  I haven't found one yet in which it was obvious to both server and
client what the right thing was.  I fear that the benefits of parallelism are
greatly exaggerated.

Instead, I've been seeing more examples in which splitting a larger task into
smaller parts, with an opportunity to do something else between parts, is more
useful.  Instead of "FETCH 5 BODY[1]" I'm seeing more cases of "FETCH 5
BODY[1]<1.32768>" "FETCH 5 BODY[1}<32769.65536>", etc.

> > I'd be rather disappointed in a database back end that had a slow EXPUNGE!
> > That would lose 95% of the advantage of a database back end.
> I can't agree with this: expunging hundreds of messages may take a
> significant amount of time, no matter what method you use for it -- even if
> for no other reason than that the server has to build and send all the
> EXPUNGE responses, in addition to whatever work is involved with
> invalidating the database records.

An EXPUNGE response is 11-15 bytes, so for 500 messages that comes to about
3K.  I doubt that there is substantial overhead in building or sending these,
unless your network link is radio (in which case you have other problems and
should do something to compress the IMAP session).

I would expect that a reasonable database would invalidate the records more or
less instantaneously, and let a background garbage collection clean them up.
I would be hard pressed otherwise to justify the disadvantages of a database
over a flat file.  I hope that people don't do databases just to do databases!

The point I'm making is that doing something in a separate session during an
EXPUNGE presupposes that there is a server which (1) allows doing it and (2)
is slow enough on EXPUNGE that it's reasonable to do it.  Yes, it's possible.
But it's a lot of extra work to do in a client, for the case of servers that
probably do not (and hopefully won't ever) exist.

> Remember that you've been doing this for a long time
> and you have more than one implementation behind you.

I don't think that makes much of a difference.

*Begin flame*
It gets awfully frustrating when I make a recommendation, and immediately
someone goes and contradicts me and/or says that it's due to a "stealth" bug
in my implementation.
*End flame*

> Oh, absolutely.  I'm suggesting an "Implementer's Guide".  Perhaps I'll try
> putting together a draft of one, and let people beat on it and add their
> favourite stumbles.  I think such a document will be useful -- because if
> this stuff were really all as obvious as you say, not so many people would
> be stumbling over it.

Thanks for volunteering.

But, what happens what you say something in your "implementor's guide" that
you feel strongly about, and I feel equally strongly that it's completely
different, and Chris Newman feels equally strongly that the right thing is
completely different from what either you or I thinK?

Most of this document will inevitably be religion.

The worst religious arguments are with folks who care only about their GUI and
not with the impact it has on anything else.  I don't know what bewilders me
more, that a single-user 32MB Pentium can be as frustratingly slow as a 640K
XT, or that said Pentium can easily swamp the net and screw things up for
everyone else.