PP13: Toward a Simpler Message Access Architecture

[Lisa Dusseault <[e-mail]>]

Begin forwarded message:

> From: Pete Resnick <[e-mail]>
>
> Toward a Simpler Message Access Architecture
>
> The architecture for the transport of mail messages embodies a core  
> principle: Messages move through the Internet simply, without  
> examination or transformation of their contents, and that they are  
> transported according to envelope information that is separate from  
> the contents. The Simple Mail Transfer Protocol (SMTP) and its  
> implementations live up to this principle to greater or lesser  
> degrees, but this is certainly the ideal that we strive for, and we  
> use it as a benchmark for SMTP design and extension. In addition,  
> when it comes to the contents of messages, the Internet Message  
> Format (IMF) and Multipurpose Internet Mail Extensions (MIME) have  
> a canonical format that is specifically independent of any  
> particular storage or user interface constraints. Again,  
> implementations keep storage and user interface constraints from  
> appearing to differing degrees, but we still strive toward this  
> platform independence in implementation and protocol development.  
> However, the structure has given us the ability to have relatively  
> complex messages (sometimes with a great deal of structure and  
> representing all sorts of complex data types) transported in a  
> relatively simple fashion. This architecture lends itself to  
> relatively simple implementation for both clients and servers:  
> Clients prepare the message object (however complex) and the  
> envelope, convert the message from any local conventions to  
> canonical format, and transmit the message using the independent  
> envelope information. End-to-end, the transport is relatively simple.
>
> In contrast to message transport, message access has not fared as  
> well in these regards. The only two message access protocols we  
> have, the Post Office Protocol (POP) and the Internet Message  
> Access Protocol (IMAP), sit at opposite ends of a spectrum with  
> regard to simplicity of implementation and operation, ability to  
> deal with complexity in message structures, and platform  
> independence. Of course, with regard to simplicity of  
> implementation and operation, POP shines: For a client to simply  
> connect to a server and download messages in their entirety, POP is  
> very straightforward. However, this simplicity pushes a great deal  
> of complexity and some computing horsepower requirements from  
> servers onto clients. Clients must have enough bandwidth to get  
> messages, enough local storage to manipulate and interpret the  
> messages they have downloaded, and enough compute power to locally  
> parse through complex message structures. In this time of smaller  
> client devices with less compute, bandwidth, and storage, this  
> makes POP use problematic. Of course, assorted extensions have been  
> added to POP to facilitate use by smaller devices, but they are  
> generally ad-hoc specialized extensions that do not address more  
> general issues for lower capability clients.
>
> IMAP on the other hand does not suffer POP's problems. IMAP servers  
> parse the structure of messages for the client and present only  
> those parts (or parts of parts) that the client requests. Recent  
> extensions have made access to and manipulation of message parts  
> even easier for lower capability clients. There is, of course, an  
> increase in protocol complexity in IMAP as compared to POP. Some of  
> that is counter-balanced by the fact that the client does not have  
> to do much of the parsing work locally, but can instead rely on the  
> server to do so. However, IMAP's central shortcoming, due to a  
> design principle that was necessary at the time of its creation, is  
> that clients have to deal with a great deal of complexity because  
> IMAP exposes much of the semantics of the server's local  
> environment to the client. IMAP was designed such that protocol  
> constructs map relatively easily to extant servers' message storage  
> models. (It should be noted that failure to have this design  
> principle at the time would have resulted in an undeployable  
> protocol.) The result is a protocol where messages are stored  
> grouped in mailboxes (which paralleled existing implementations'  
> file storage structure), much of the ancillary data associated with  
> messages is stored in the mailboxes themselves (requiring a small  
> set of data), and operations may be either synchronous or  
> asynchronous in nature depending on the particular operation. These  
> design choices introduce interesting and unexpected implementation  
> complexities on clients.  IMAP's support of a hierarchical store of  
> messages (as against POP, which has a single flat spool of  
> messages) means that even if a client only wants to present a  
> simple one-dimensional view of messages, it still must support  
> IMAP's hierarchical store in order to discover all of the messages  
> on the server. In particular, in order to discover changes to the  
> mail store, clients must SELECT those mailboxes in which the client  
> has interest and can not get information about changes to arbitrary  
> parts of the store. With regard to the data storage model, clients  
> that perform operations outside of IMAP's architectural model that  
> might need to store and retrieve ancillary data with messages to  
> support those operations (for instance, a particular messages  
> display size and screen positioning) must either rely on the server  
> to implement such functionality (which is not a requirement for  
> servers) or must store that information locally (which is  
> infeasible for some clients). Finally, the mixture of synchronous  
> and asynchronous operation sometimes makes for tricky  
> implementation choices. Of course, there are the well-understood  
> examples of certain command sequences that cannot run concurrently,  
> but restrictions on untagged EXPUNGE responses also make  
> implementation difficult.
>
> Server technology has come a long way in the past 20 years. In  
> particular, scalable multiple-access database storage  
> implementations are now widely available and in use. Not only does  
> this mean that arbitrary data can be easily stored and retrieved  
> (including by multiple simultaneous users), but it also means that  
> hierarchical file structure needn't be maintained by the  
> application in order to obtain decent performance. Use of such a  
> system as a backend for a message access protocol simplifies design  
> significantly and greatly aids flexibility. I suggest that a new  
> message access architecture could maintain all of the advantages  
> that IMAP has over POP, yet remove some of the difficulties  
> associated with reflection of underlying design constraints in  
> protocol considerations.
>
> What follows is a rough outline of such an architecture. It maps  
> onto current technologies in such a way that transition to the  
> architecture would be more straightforward, sometimes at the  
> expense of simplicity. (Consider this a target at which to shoot  
> rather than a fully-fleshed out solution. In particular, neither  
> access control nor end-to-end security issues are discussed in this  
> proposal.)
>
> 1. A message access store will contain three sorts of objects:  
> message objects, multipart objects, and terminal objects. All  
> objects have attributes. The objects are comprised as follows:
> 	a. Message objects must contain one multipart or one terminal  
> object. Each message must have a "unique identifier" attribute  
> which is unique within the message store. A message will normally  
> also have a series of attributes that contain message source and  
> transport information. (This would normally consist of things  
> normally found in RFC 822 headers. However, it will not include  
> MIME information.)
> 	b. Multipart objects are an ordered collection containing one or  
> more message, multipart, or terminal objects. Each multipart will  
> have a type attribute, indicating the type of collection, and may  
> have attributes corresponding to MIME disposition, description, or  
> identifier information.
> 	c. Terminal objects contain data that comprises the object.  
> Terminal objects have a type attribute, indicating the type of data  
> contained in the object, and may have attributes corresponding to  
> MIME disposition, description, or identifier information.
>
> All of the attributes mentioned above are read-only. Message store  
> objects may also have writeable attributes associated with them.  
> These are talked about below with regard to message access clients.  
> In any event, these attributes are for clients only. Message access  
> servers do not add or change them.
>
> Multiple objects may contain a given object. The deletion of a  
> container object only deletes once instance of a shared contained  
> object.
>
> A message access server will likely house several message stores.  
> These message stores may share data, in which case unique  
> identifiers must remain unique and object instances must be  
> maintained across all message stores.
>
> 2. A message access client may perform the following functions with  
> regard to the message store:
> 	a. Authenticate itself to the message store server.
> 	b. Search for messages with particular attributes, receiving a  
> list of message identifiers. (The client may request that the  
> results of searches be sorted by any particular attribute. The  
> client may request to be returned only a particular portion of the  
> search results.)
> 	c. Retrieve any message or object contained therein (or portion  
> thereof).
> 	d. Retrieve the attributes of any message or object contained  
> therein (including a list of objects contained in the message or  
> object, recursively if desired).
> 	e. Store a message (allowing that message to contain parts already  
> in the message store).
> 	f. Store or change writeable attributes. (Writeable attributes are  
> arbitrary in nature; that is, an arbitrary attribute name may be  
> used and it may be given an arbitrary value. Although there may be  
> a registry of such attributes, they are strictly for client use.)
> 	g. Given an attribute, search for all unique values of that  
> attribute across the message store.
> 	h. Delete a message.
>
> All client actions are sent and completed asynchronously. Responses  
> to client actions are tagged to indicate the order in which they  
> were completed. A client will receive responses regarding storage,  
> deletion, and changes of objects and attributes even if the  
> particular client in question did not initiate those actions.
>
> It is imagined that a message delivery agent is just another  
> message access client.
>
>
> -- 
> Pete Resnick <http://www.qualcomm.com/~presnick/>
> Qualcomm Incorporated - Direct phone: [phone], Fax: (858) 
> 651-1102