[Jmap] blob API in draft-ietf-jmap-{core,mail}-01

[Josef 'Jeff' Sipek <[e-mail]>]

I finally got a bit of time to go over the blob interface in the draft.  In
general, I like the idea of a separate endpoint to get/put raw emails.  The
other uses of this blob interface are what I think needs more work / details.

The blob upload scheme for attachments is a bit scary - not only are we
dealing with email (which is complex enough), but the draft pulls in
generic-ish object storage and deduplication along with the associated
issues and complexities.


Downloads
---------

jmap section 4 says:

   After completing authentication, the client will receive a
   _downloadUrl_ as part of the response.  This is in [RFC6570] URI
   Template (level 1) format.  The URL MUST contain variables called
   "accountId" and "blobId".  The URL SHOULD contain a variable called
   "name".

Why is accountId required in the URL?  I don't see a reason why a server
couldn't use blobIds that either have internal structure to correctly
determine the accountId, or to not care about accountId at all.  Concerns
about malicious users accessing known hashes can be addressed by salting the
hash when generating the blobIds on the server.  This is of course an
implementation detail, but I think it demonstrates that the accountId seems
superfluous.  I'd expect the spec to say that the URL MAY contain accountId,
just to make sure that clients expect it.


Uploads
-------

jmap section 5.1 (201: File uploaded successfully) says:


   o  *expires*: "Date" The date the file will be deleted from temporary
      storage if not referenced by another object, e.g. used in a draft.

Should the spec provide some guidance how far into the future the expiration
date should be?

Should it explicitly point out that the timestamp is purely advisory, and
the blob may disappear at any time for a variety of other reasons?

jmap section 5.1 (201: File uploaded successfully) also says:

   If identical binary content is uploaded, the same _blobId_ SHOULD be
   returned.

If the same blobId is returned, should the expiration date be bumped as if
it were the first upload of the blob?

jmap section 5.1 (201: File uploaded successfully) also says:

   Once the file has been used, for example attached to a draft message,
   the file will no longer expire, and is instead guaranteed to exist
   while at least one other object references it.  Once no other object
   references it, the server MAY immediately delete the file at any
   time.

Once a blobId is used by an object (e.g., via setMessages), is it supposed
to be still accessible via the same blobId?  Or can the server move the
data associated with the blobId out of the "pending blob" storage into a
more permanent place and in the process generate a new blobId?

I'm imagining the "pending blob" storage to be a simple directory with one
file per blob, but once a setMessages tries to take a "reference" on the
blob, the server instead synthesizes an RFC5322 email and stores it in the
requested mailbox and then removes the "pending blobs" copy.  It would be
nice if jmap worked seamlessly on top of maildir/mbox/etc. without the need
for too many extra storage pools or duplication of data between the blob
store and the actual mailbox storage.


Eviction
--------

The expiration-based eviction is ok, however I think there needs to be more
clarification about the other causes for blob eviction and how eviction
interacts with the rest of jmap.

For example:

jmap section 5.1 (201: File uploaded successfully) says:

   If uploading a file would take the user over quota, the server SHOULD
   delete previously uploaded (but unused) files before their expiry
   time.

jmap mail section 5.3.1 (Saving a draft) says:

   If any of the files specified in _attachments_ cannot be found, the
   creation MUST be rejected with an "invalidProperties" error.

When these two behaviors are combined, it is easy to run into a ugly
scenario where a client wants to save a draft with two large attachments
with quota enabled.  For example:

(1) the user is 1.5MB away from reaching quota, there are no unreferenced blobs
(2) the client uploads first attachment that is 1MB in size
    (a) the user is now 0.5MB away from reaching quota
    (b) there is one unreferenced blob (the one just uploaded)
(3) the client uploads second attachment that is 1MB in size
    (a) since the user would be over quota, the server evicts the previously
        uploaded blob to make space for the new blob (jmap 5.1)
    (b) the client gets no notification that the previous blob was evicted
    (c) the user is still 0.5MB away from reaching quota (0.5MB + 1MB - 1MB)
    (d) there is one unreferenced blob (the second one uploaded)
(4) the client tries to store the actual draft mail, referencing the two
    blobIds obtained from the uploads
    (a) the server fails the setMessages call because one of the blobIds
        doesn't exist (jmap mail 5.3.1)

In this case, the client uploaded 2MB+ and accomplished absolutely nothing.
A naive client would then proceed to reupload the missing attachment, using
up even more bandwidth but still making no actual progress.

I think it would be good for jmap all requests to either succeed or fail in
a way that tell the client what can be done to get more forward progress
(even if it is sub-optimal) or that there is no way forward.  In this
particular example, the issue is that the invalidProperties error returned
from setMessages covers multiple situations.  At least some of them are:

(1) client error: the client didn't upload the attachment before trying to
    reference it
(2) blob expired: the server removed the blob because it remained
    unreferenced for too long
(3) blob evicted: the server removed the blob early because of over-quota
    situation, etc.

There is nothing to indicate that repeating the upload will or will not
result in any progress.


By the way, a number of the HTTP status code descriptions include: "There
is no content in the response."  Is that saying that the response body
SHOULD be ignored?  Or that the responses' Content-Length MUST be 0?

Anyway, that's it for now.

Jeff.