Re: [Jmap] simpler future release & unsend without outbox (was Re: Best vs Good enough - adoption of JMAP)

["Chris Newman" <[e-mail]>]

On 27 Apr 2017, at 22:46, Neil Jenkins wrote:
> On Thu, 27 Apr 2017, at 02:06 AM, Chris Newman wrote:
>> Well, if you believe that and I don't believe that,
>
> Then clearly we need to discover where the divergence in mental models
> is occurring.

Here's my guess at the root cause:
* You're trying to mirror a particular client UI model in the mail store 
server design.
* I'm trying to minimize the necessary server infrastructure to support 
popular UI models (including yours), particularly when the server is RFC 
4865 standards-based already.

I see the following divergence as a result of these perspectives:

1. I believe there are four message types involved in this discussion:
   * draft messages
   * client-side outbox
   * future release RFC 4865 submitted
   * other submitted messages
I believe we agree the protocol should make it simple to separate and 
count these classes of messages for UI display. However, I strongly 
prefer a design where the latter two message types are treated the same 
on the server because it reduces the necessary server infrastructure 
(and, in particular, it avoids conflating mail store and mail queue 
semantics).

2. I believe that mixing client-side outbox and future release messages 
is a poor UI that will confuse users. Client-side outbox messages have 
not been submitted and will not move forward if the client is offline or 
the client removes the message from the outbox. Future release messages 
have been submitted and will move forward when the client/MUA is 
offline, even if the message is removed by the client when offline. 
These messages can only be stopped by an explicit online "unsend" MTA 
operation. This is why I believe mixing these messages will be 
confusing. You apparently do not agree with me on that point. As with 
all UI design, at some point there will be disagreements between 
reasonable people that are not resolvable. I suspect this is one of 
those cases. To me that suggests the resolution is a protocol that 
doesn't constrain the UI choices in this area.

3. I view "submit" and "unsend" as MTA actions that should not be 
confused or conflated with mail store operations. I'm opposed to 
protocol designs where a message move, delete or flag changes triggers 
MTA actions as a side-effect. Instead, I propose JMAP have explicit 
"submit" and "unsend" commands that first perform the MTA action but can 
also trigger mail store actions (like move message or flag change) if 
the MTA action succeeds.

4. I believe the "unsend" MTA operation can be applied to any submitted 
message, not just future release messages. It's more likely to succeed 
on a future release message, but there are plenty of proprietary systems 
that allow unsend to work at least until the message has been seen by 
the recipient so we shouldn't disallow that behavior in the JMAP 
protocol design.

>> * Every server that supports future release and unsend will provide
>>   the same interface to the client. No need for the "always empty
>>   outbox" alternative server behavior that you'll see with the outbox
>>   model.
>
> The server will have to advertise whether it supports scheduled send
> regardless of the mechanics underneath. For servers that do not 
> support
> scheduled send, I agree that an always-empty outbox mailbox may feel a
> little superfluous, however the point is to, as you say, always 
> provide
> the same interface to the client regardless. The client will know from
> the capabilities that it must always be empty, so will probably hide 
> it
> in the sidebar or equivalent navigation mechanism (in fact I expect
> clients will probably hide the outbox when empty regardless of 
> scheduled
> send support).

Our server already supports scheduled send via RFC 4865 and a 
proprietary 'unsend/recall' mechanism. However, with the server-side 
outbox model we'll not expose those capabilities to JMAP clients because 
our mail store has no access to mail queues and that is an intentional 
and deliberate design choice for security and scalability reasons.

I've proposed a design where existing RFC 4865 standards based servers 
can transparently expose those capabilities without constraining 
possible MUA designs.

>> * This does not unnecessarily constrain server implementation, and
>>   permits simpler server designs.
>
> I do not believe this to be the case, as I will explain below.

If an "outbox" is a mail queue (which it is, by definition, if there is 
a daemon that accesses a message from the outbox and passes it forward 
in the delivery system), then a server-side-outbox unnecessarily 
constrains server design.

>> * This preserves clean mail store and mail queue separation including
>>   the security and operational benefits that provides.
>
> This is I think where the confusion lies. The outbox really isn't a 
> mail
> queue, and no one is suggesting you merge it with your mail store.
>
> You clearly have existing infrastructure that supports holding 
> scheduled
> messages inside your mail queue and possibly cancelling them from 
> that.
> You wish to reuse this infrastructure, of course! That's fine. To do 
> so,
> when the message is moved to the outbox, you would *also* submit it 
> into
> your mail queue. The copy in the outbox would be the representation in
> JMAP land, not part of your mail queue.

This implementation approach includes properties of the following 
anti-patterns: "database-as-IPC", "race hazard" and "action at a 
distance". https://en.wikipedia.org/wiki/Anti-pattern

It raises questions like this: Should the shadow copy in the outbox have 
the received header that is present on the message in the submission 
queue? What happens if an MUA deletes the copy of the message in the 
server-side shadow outbox and the link between MUA and mail queue is 
down when that action is performed? Does the delete fail with a special 
error or does the delete succeed and then generate a new DSN-like 
message saying the attempt to unsend that message failed? Are both of 
those behaviors allowed?

Anyway, this has reinforced my point about putting unnecessary 
constraints on server design. This proposal requires complex linkages 
between the JMAP server, the mail store and the mail queue (creating a 
three-party system with object aliasing that will have unintended 
consequences).

On the flip side, my proposal was neutral on the matter of client UI. If 
your client wants to create a virtual outbox view for already-submitted 
future release messages and offer the unsend operation only on those 
messages, that's fine.

> When the message is sent to the
> next hop and can no longer be recalled,

These two things don't necessarily happen at the same time. For our 
implementation, a message can be recalled as long as the message lives 
in a queue in one of our MTAs. It can pass to the next hop MTA and still 
be unsent. Many proprietary systems allow message recall until the 
message has been seen by a recipient. JMAP's design shouldn't disallow 
that.

> you then move the JMAP message
> from outbox to sent, and remove the `\Draft` keyword.

Why require the addition of a new signaling infrastructure from MTA 
queues to mail store and add new churn to the mail store when the same 
MUA UI can be achieved without physically moving the message between 
folders at the future release time?

> Other systems may choose to implement it differently, not passing it 
> to
> the mail queue until the scheduled time is reached. As has been 
> pointed
> out already, many systems such as Gmail, FastMail etc. already have
> infrastructure to do this kind of thing since it is also used for
> deleting messages that are X days old from the trash, supporting 
> snoozed
> mail, etc.

Actually, my proposal makes this server design approach simpler. There's 
just an internal flag in the sent folder to indicate if the message has 
been passed forward. The mail queue daemon that operates on the mail 
store sent folder, can transfer the message to the next hop MTA and set 
that flag at the future release time. No need to move the message 
between folders.

> The outbox protocol is agnostic to how you wish to implement this. 
> Your
> proposal constrains you to only the option *you* already have
> implemented.

My proposal makes more server approaches possible, while being agnostic 
from the client UI side (clients can present a virtual outbox for future 
release messages or not as they choose).

>> * This makes it clear when the message enters the submission system.
>>   There's no confusion as one has with an outbox model where a 
>> message
>>   in the outbox may be submitted (if it's on the server) or maybe not
>>   (if the client is offline when the message moves to the outbox).
>
> This is clearly up to the client to distinguish, but I strongly 
> disagree
> this is a confusion. For the user, there is the outbox, which 
> represents
> the messages that are yet to be sent. If they open the outbox, I 
> expect
> most MUAs would label why they have not yet been sent (e.g. waiting 
> for
> internet connection, scheduled for time X).

See 2 above.

>> * Submission can have semantics fully compatible with existing
>>   infrastructure, making this easier/faster to deploy.
>
> As mentioned above, a lot of systems already have infrastructure for 
> "do
> action X in folder Y when time Z is reached", or you can alternatively
> use your SMTP-queue-scheduled-send to implement the outbox model, so
> this is actually *more* compatible with existing infrastructure than
> your proposal.

It's not "my SMTP-queue-scheduled-send". It's the IETF standards track 
scheduled send mechanism, RFC 4865.

>> * The "outbox" model constrains unsend to only work if the message is
>>   in the outbox. This model has no such constraint and works the same
>>   regardless of where a message that can be "unsent" lives in the
>>   infrastructure.
>
> When composing and sending a message, it goes through 2 or 3 states.
> These need to be represented in the mail store (and thus the protocol)
> so that users can understand what is going on, and what actions are
> available to them.
>
> Two of these we (I believe) agree on, since they are from IMAP:
>
> (1) The message is still being composed. It SHOULD be in the
>     "drafts" mailbox and MUST have the "\Draft" keyword. This is the
>     same as IMAP.

Agreed.

> (2) The message has been sent, cannot be recalled.

These two statements are not synonyms; see my item 4 above.

>     It SHOULD be in the
>     "sent" mailbox (although services may provide means for users to 
> ask
>     it to go somewhere else, for example we have a few users that 
> prefer
>     their sent mail to in the same mailbox as the message they were
>     replying to).
>
> The third (new) state is for "scheduled" messages. We have two options
> for how this is represented:
>
> 1. By its presence in the "outbox". Moving it out is therefore the
>    logical thing to unschedule it.

I disagree. Unsend is a separate operation with different success/fail 
consequences from the move message operation. I object to confusing 
these two operations; see my item 3 above.

> 2. By a keyword. This permits it to be in any mailbox, although we 
> would
>    probably add an outbox mailbox to our server for holding these
>    messages by default. Removing the keyword would probably be a 
> logical
>    way to unschedule it. This option is quite a bit less efficient at
>    the protocol level, as without a Mailbox object (which has a
>    precalculated total count), the client will have to issue a search
>    every single time there's a state change to see if there are any
>    scheduled messages.

I disagree. Unsend is a separate operation with different success/fail 
consequences from the change flag/keyword operation. I object to 
confusing these two operations; see my item 3 above.

My proposal is to have future release sent messages tagged with their 
future release date. The MUA can create a virtual folder for future 
release messages with a future date. Whether the server caches a count 
for the messages in that virtual folder is a server design optimization 
choice. If enough clients request that count that it helps server 
scalability to cache that count, then servers will cache that count. The 
count of future release messages is thus not relevant to the issue under 
debate.

> I'm presuming you are not advocating for option (3), which is no
> representation in the mail store, since this would be terrible for the
> user experience, and your proposal explicitly said "Add a way to
> search/select messages in the Sent folder that may be possible to
> unsend", which would indicate there must be some kind of state
> representing this status.

Per my item 3 above, I view "submit" and "unsend" as MTA actions. There 
are two ways to reflect successful MTA actions in the mail store:

A. When the submit action succeeds, the message is moved from draft 
folder to sent folder, the \draft flag is removed (if set), and an 
envelope annotation (including future release date if there is one) is 
added if not already present. When unsend succeeds, the message is moved 
from the sent folder back to the draft folder. The unsend operation can 
be applied regardless of whether the message is a future release 
message.

B. When the submit action succeeds, the \draft flag is removed, a 
$submitted flag is set and an envelope annotation (including future 
release date if there is one) is added if not already present. When the 
unsend succeeds, the $submitted flags is removed and a \draft flag is 
added.

I slightly favor option A, but have no strong opinion on this matter. 
It's even possible to design the submit & unsend actions so both of 
these are possible (always make the flag change and make the 
folder-move-on-success an optional feature of the MTA actions).

> So however you implement it, you need to trigger some state change on
> the JMAP server for each logical step so that clients can represent
> this, and update its local model as changes occur. The choice of
> representation is mainly down to what will map more easily to how
> clients wish to represent the set of scheduled messagese to users.

I disagree. The protocol design has peer goals on this matter: make 
popular UI paradigms straightforward and minimize necessary server 
infrastructure.

> UX designers seem to have overwhelmingly decided that putting these
> scheduled messages in an outbox rather than mixing them in with sent
> items or elsewhere is a clearer model for users to understand. I
> provided 5 example products in the previous email that do this, but if
> that's insufficient I noice we have representatives from Apple, Google
> and AOL on this list; perhaps they can ask their respective UI teams 
> to
> comment on what they would recommend?

I do not contest that client-side outbox is a popular UI model. I 
further agree the protocol design should allow the presentation of a 
server-side outbox UI. So I don't think there's a point of contention 
here.

		- Chris