[Jmap] new JMAP server for prototyping

Jamey Sharp <jamey@minilop.net> Fri, 14 May 2021 18:33 UTC

Return-Path: <jamey@minilop.net>
X-Original-To: jmap@ietfa.amsl.com
Delivered-To: jmap@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5D2733A3C04 for <jmap@ietfa.amsl.com>; Fri, 14 May 2021 11:33:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=minilop.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VmR3AP5Pd5oq for <jmap@ietfa.amsl.com>; Fri, 14 May 2021 11:32:56 -0700 (PDT)
Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3F5EC3A3C00 for <jmap@ietf.org>; Fri, 14 May 2021 11:32:55 -0700 (PDT)
Received: by mail-il1-x133.google.com with SMTP id p15so568262iln.3 for <jmap@ietf.org>; Fri, 14 May 2021 11:32:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=minilop.net; s=google; h=mime-version:from:date:message-id:subject:to; bh=WGBymLfv5NgvyiMkK23/jePjPNO/DPyqlmmn1tC5Y8w=; b=JT/t+df8iRHV8ocSRsWBlAkHjlsD0Rf2vThgWlSWaho5LjF2d5o958jzJdxg5OprtN ZcofH4XzcsTKcPszqMijwtWm3vEyrW3l/X9u4EBXQuRTYOz/Y6FkrKMvgsVarQC7NsI1 x7aaOlbYLdq8QaE9IkvvizN+RxJQhyXCueugg=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=WGBymLfv5NgvyiMkK23/jePjPNO/DPyqlmmn1tC5Y8w=; b=q2cASyvA2E07hC++tGjzCVVYplc2mLBqZ1s41urnNjt1CiItYGfIV2ffLW0myqPV/F VJr5eRclyIFz+iX9W1eYj2QwqpemEpeg2lRo8YiskGhXPBWLJVIFVj2yksILXHgqoDGu Ld4BoM6QvsrzyV9iyJ0qUfgLZwXbgJ6fcYY/w5JGaBC6EPTFC1mM9I2I8liZe4TYvF3J ZB3A9e+NHk9ETu0krtNFJbU6di1uHG36tLi9h+nnqNeBWXrXOxbWT6bqXyMeB18EG6fx EiFxJxKeuOSNtFGs76QMD+pVh19EYYSBbY3gPHHKLM2N5Lk8PC2yFA4MhkdBZf6mSmJX qzWA==
X-Gm-Message-State: AOAM5325ZQw0/KZIWYkunQ/RugXcaKTGWmrn0xSR0NmftLFX1oFOG+es xj3UKBlIFiziYDMedZQEZg+0q6oMGl9kTlpfvdZw9CHKRNk=
X-Google-Smtp-Source: ABdhPJzBQjGhNS7W0mPn94bxQQ+G3INBmTtwq7bq4GhJQ//kxG0mNR36idY22SzGGLnqCO6BrrEwv8LMxs0jZ+rmn78=
X-Received: by 2002:a05:6e02:671:: with SMTP id l17mr41562903ilt.267.1621017174221; Fri, 14 May 2021 11:32:54 -0700 (PDT)
MIME-Version: 1.0
From: Jamey Sharp <jamey@minilop.net>
Date: Fri, 14 May 2021 11:32:42 -0700
Message-ID: <CAJi=jaeZ7uC+CoXLhSekvO-pOJnHgSOFNzZ_WBCv1Sf9HfqK2A@mail.gmail.com>
To: jmap@ietf.org
Content-Type: multipart/alternative; boundary="000000000000d84bfa05c24e786d"
Archived-At: <https://mailarchive.ietf.org/arch/msg/jmap/b7X6d2ZdvdFGWIR7AJ9MUJ9nwYk>
Subject: [Jmap] new JMAP server for prototyping
X-BeenThere: jmap@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: JSON Message Access Protocol <jmap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/jmap>, <mailto:jmap-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/jmap/>
List-Post: <mailto:jmap@ietf.org>
List-Help: <mailto:jmap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/jmap>, <mailto:jmap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 14 May 2021 18:35:41 -0000

Hey all: for the past few weeks I've been working on a new JMAP server with
the goal of making it easy to prototype new data models. I'd love it if
folks could take a look and maybe help implement or at least validate it.

https://github.com/jameysharp/jterritory

I went down this rabbit-hole after trying to figure out how to prototype a
feed-reader data model based on JMAP. I previously wrote up rationale for
that here:

https://jamey.thesharps.us/2020/08/06/feed-reader-cache-coherency/

Implementing JMAP correctly is pretty complicated, and while I do think
it's all necessary complexity, that makes it difficult to explore new
application domains. The core idea for this project is to not worry too
much about performance, but allow a developer to specify just a schema for
each data type and optionally some filter and sort extensions, and get the
standard methods for free.

I've been confused by a few things in RFC8620. I was only going to give a
couple examples but ended up with a complete list of everything I remember
having trouble with, so I hope it works out to just dump it here. I have
some more notes afterward on my current implementation and plans.

- Should creating multiple objects allow circular references, or must it be
a DAG of new objects? The former seems hard to get right if any of the
creations fail.

- Can result references appear in nested objects within a method's
arguments, or only on the arguments directly within the top-level
Invocation?

- Does the client need to ensure that it never uses a creation ID of "foo"
if there's some random string whose value happens to be "#foo", or does
/set need to know which properties have Id type?

- If the client uses the same creation ID in two method calls, despite the
"SHOULD" cautioning against it, and the second create fails, should the
first one's ID continue to be used? Is "the most recently created item with
that id" the most recent attempt, or the most recent success?

- I think I get it now, but I've been confused about how the responses from
/copy relate to its three phases. There's one /copy response covering
phases 1 and 2, then optionally a single /set response for phase 3, right?

- What value should the position attribute in a /query response have if the
requested position is past the end of the results?

- If a client has a sparse array of query results and gets a /queryChanges
response indicating that it should delete an object that it didn't have in
the array, then I don't see how it can tell whether to shift down any of
the objects it does have: the deleted object could have been anywhere. This
interface only seems safe so long as clients either always keep a complete
prefix of a query's results, or invalidate their cache if told to delete an
object they didn't know about.

- What does property immutability have to do with query result changes?
Isn't the only thing that matters whether the property actually changed,
rather than whether it could change? Is it just that some reasonable
implementations can't compute certain changes for mutable properties?

- The specification for upToId seems to imply to look up the position of
the given ID in the new results, though that isn't entirely clear to me.
But I think it makes more sense to find its position in the old results,
and send changes that update the client's cache to the same length. That
means the optimization still works even if the selected object is no longer
in the results, and puts an upper bound on how much data the client might
have to deal with: twice the number of items it has cached already.

But overall I've found the spec pretty clear and solid so far.

In the rest of this note I offer some more details on my current status and
approach.

I'm taking the simplifying assumption that there are no dynamically
computed properties; a call to /set has to update any related objects at
the same time. That lets me keep all objects in a single database table,
storing simple change-sequence numbers alongside raw JSON, instead of the
complexities recommended for email implementors. There are performance
costs to this simplicity, but that's a deliberate trade-off I'm making.

At this point I have the semantics for get, changes, set, and most of query
implemented, maybe even correctly. I've also implemented result references
and Core/echo, which go shockingly well together for testing. I haven't
hooked anything up to HTTP yet, just writing tests that pass JSON in to the
underlying functions.

Currently all my tests are property-based randomized testing. I'd like to
think my tests might be interesting for other server implementors. If there
are test suites I might be able to reuse, I'd love to hear about them.

My observation about implementing queryChanges is that you only need to be
able to report changes against queries which have previously actually been
run, so caching returned query results is enough to do the job generically,
without understanding the semantics of the data model or filters. Of
course, caching also helps with the fact that without that understanding, I
also can't provide indexes to make filters perform well. Refreshing the
cache can be as fast as calling /changes, as long as the cache includes all
the selected sort keys for each object. Also, I think I can enable
prototypes built on this framework to specify custom indexes, presumably
after the developer has gotten some experience with which filters are most
frequently used.

I've yet to tackle the session state, blobs, or push, but those don't seem
like they'll have hard questions about semantics, just some possibly tricky
engineering choices. We'll see how that goes.

Thanks for a nice spec and for getting to the end of this message. 😅
Jamey