Re: [Jmap] Best vs Good enough - adoption of JMAP

[Bron Gondwana <[e-mail]>]

On Wed, 26 Apr 2017, at 14:49, Chris Newman wrote:
> On 25 Apr 2017, at 16:46, Neil Jenkins wrote:
> > I think there's been some misconception over the intended semantics of
> > the outbox proposal. The outbox was never intended to be the 
> > submission
> > queue, with mail queue semantics. It is intended as a standard 
> > mailbox,
> > with mailstore semantics, because it is where messages go **before**
> > they are passed on to submission. The server would validate the
> > message/envelope at the time of moving it to the outbox to ensure it 
> > can
> > be accepted for submission later of course. If you have existing
> > mailstore + SMTP infrastructure, implementation is not much more than
> > having some small demon that triggers to take the message from the
> > mailstore and submit it to the SMTP queue at the appropriate time.
> 
> And as soon as you add that "small daemon" (also known as a mail queue 
> processing daemon), you've now required the mailstore to also have mail 
> queue semantics. This includes key semantics such as: the mail queue 
> daemon has full read / delete access to those messages even when the 
> user is offline. If the delivery doesn't happen when the user expects, 
> the user is going to make a support call, so we now need all the 
> monitoring, logging and robustness that comes with a mail queue. And 
> addresses that were valid at the time the message entered this mail 
> queue can become invalid later, so there has to be bounce-processing for 
> when that happens, etc, etc.

Disagree on bounce processing.  It's no different than the next SMTP hop in that regard.

It's exactly the same as other scheduled stuff we already support in our server such as calendar alarms based on calendar events - which we happen to store in the same backend - so it already needs event scheduling.

I even blogged about our plans

https://blog.fastmail.com/2015/12/25/a-ghost-of-fastmail-future/

(complete with over-enthusiastic date predictions)

But adding the message to the Outbox also generates a full logged, replicated, replayable event in the #events mailbox which will be processed by a single queue daemon per server, and which will be monitored in the same way we monitor all our queues - by injecting messages at the start and making sure they are flowing out the other end.

> The only way to have a server-side outbox that is not a mail queue would 
> be if messages sit in the outbox indefinitely unless the user is online 
> and their client decides to issue a submit command for a message in that 
> fictitious outbox.

Yeah, that makes delayed outbox mostly pointless.  Particularly in the multi-client world.

> > From a user experience point of view, this model gives us an
> > understandable conceptual place for messages that are *going* to be
> > sent but have not yet been picked up by the postman, to use a real
> > world analogy.
> 
> This analogy only works with a desktop MUA where the "outbox" is on the 
> desktop and thus owned by the "postal customer" (and incidentally, the 
> outbox won't empty if the desktop is shut down). The hardware running 
> JMAP is not owned but the "postal customer"; it's effectively owned by 
> the postman. So an "outbox" on the JMAP server means the message has 
> been given to the postman already; the postman is just waiting for the 
> right time to release it to the next hop.

We could imagine it in the case where there's a company "postman" who's still in the building for a pre-determined amount of time before they take the packages down to the central processing facility, and while it's still in the building you can waylay the postman and ask for it to removed from the postbag...

> > Because it is a standard mailbox, we automatically have
> > an API for the client to fetch and show to the user all the currently
> > scheduled messages. I think this is important functionality as a user
> > if we have delayed send (and of course we also have the ability to 
> > then
> > cancel any of the future submissions, by just moving the messages back
> > out of the outbox).
> 
> The problem with server-side-outbox in JMAP is that it constrains the 
> server implementation to three options:
> 
> 1. The mail store includes a mail queue for each user.
> 2. The mail queue, at least for delayed messages, can be accessed with 
> full JMAP semantics. Including things like body structure, mod sequence, 
> flags, etc.
> 3. The JMAP outbox is always empty.
> 
> I really want a fourth option for server implementation:
> 
> 4. The mail store and mail queue are completely separate. The mail queue 
> daemon never accesses the mail store. The set of actions JMAP performs 
> on objects in the mail queue is limited to only those that are strictly 
> necessary to achieve a useful goal (envelope extensions like future 
> release at submission time, unsend for messages still in local queue, 
> and status feedback tagging on sent messages).

You appear to be proposing one of:

a) there is no support for delayed sending or undo
b) the user has no visibility or control of queued messages after submitting them
c) the user can control queued messages, but needs to use a different protocol than JMAP to do it
d) a new object type for queued messages needs to be defined in JMAP, with operations that allow achieving the above useful goals.

If (d), are you willing to write up the object definitions and client advice for how they would be used, so they can be tested?

> > With my client author hat on, I like this model because it makes it
> > easier to implement full offline support. If I am offline, the outbox
> > represents messages waiting to go out from the client. From an
> > implementation perspective it is much easier if my offline sync code
> > doesn't need a separate path for keeping track of messages to be sent
> > vs. normal messages moved to another folder.
> 
> A client-side outbox makes sense to me. It's a server-side outbox that I 
> find problematic. Let me ask you this:
> 
> Suppose JMAP implements a server-side outbox (aka mail queue). The JMAP 
> client synchronizes the server-side outbox and goes offline. While the 
> client is offline, the message gets submitted and the user decides they 
> don't want to submit the message and delete it from the local cached 
> outbox. Now what? Have you really simplified things by adding a 
> server-side outbox to the mix?

When the client re-syncs, it sees that the message it tried to move back out to Drafts no longer exists, but there's a new message in Sent.  The user notices that the send happened, oh well.

This is the same as if you hit "undo" in the Gmail interface after sending a message, but unfortunately your internet connection dropped for 30 seconds, and it was too late.

Clearly the client can't know for sure that the undo succeeded until it gets confirmation from the server.  In functional programming parlance, moving messages in and out of the "Outbox" folder is impure - it has side effects.

> > The *only real difference* between a "sendMessage" command and using
> > "move to outbox" as the equivalent command, is that clients can get
> > the list of "delayed" messages waiting for submission. I think this is
> > valuable, and pretty much required for "scheduled send", although not
> > so important for "undo send". The fact that this is really the only
> > difference is why servers that can't or don't want to implement
> > delayed send or expose messages waiting to be sent can use the "empty
> > outbox" model.
> 
> My product has already implemented future release, already implemented 
> unsend from local mail queue and implemented message tracking so we can 
> get status of a message in the local queue (and potentially beyond 
> that). I'm willing to expose these in JMAP if the protocol is designed 
> in such a way that we preserve clean functional separation between mail 
> store and mail queue. I'm very much not interested in all the complexity 
> a server-side outbox adds on top of those three features.
> 
> I'd argue that "scheduled send" is a mail client feature and "future 
> release" is a mail server feature. If there's a daemon that can move the 
> message to the next hop even when the client is offline, you're talking 
> about "future release" and the message has semantically been submitted 
> already. So I think JMAP should put "future release" messages in the 
> Sent folder to avoid confusion with a client-side outbox. I'd like JMAP 
> to include a way to "unsend" future release messages so JMAP clients 
> have access to the unsend feature we've already implemented.

If you could write up how, over JMAP, a client could tell that a message was future release and interact with its queue, that would probably help everyone understand how what you're proposing would work.

Thanks,

Bron.


-- 
  Bron Gondwana
  [e-mail]