[Jmap] Hacker News Discussion - getMailboxes single data set vs tree primitives or paging

Bron Gondwana <brong@fastmail.fm> Sun, 07 May 2017 11:06 UTC

Return-Path: <brong@fastmail.fm>
X-Original-To: jmap@ietfa.amsl.com
Delivered-To: jmap@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CF6C51279E5 for <jmap@ietfa.amsl.com>; Sun, 7 May 2017 04:06:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.02
X-Spam-Level:
X-Spam-Status: No, score=-0.02 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=fastmail.fm header.b=GXv4NabP; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=PznuiJDB
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hva_-ZEodq8D for <jmap@ietfa.amsl.com>; Sun, 7 May 2017 04:06:54 -0700 (PDT)
Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5358912751F for <jmap@ietf.org>; Sun, 7 May 2017 04:06:54 -0700 (PDT)
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id C611C207C7 for <jmap@ietf.org>; Sun, 7 May 2017 07:06:53 -0400 (EDT)
Received: from web4 ([10.202.2.214]) by compute6.internal (MEProxy); Sun, 07 May 2017 07:06:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.fm; h= content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=ejEInF/oIZxv7U7gX9AlaXmjytYaUdccqrz3Z+CKqTk=; b=GXv4NabP Tzjr2ZvB961D2zQWc03FkX4FXcUFuj1hAWZOUTWYzFR8gAt5whynHUA6CkDeoM4q wn41Tkaw5kqNvi3eRqQEFWxxZkYQWKeoYnbkCAqJ/7A0rllrxvZPxWtpCkRVHAG8 l3At8n96SZ7lWQLnCOyDm5af7GnNAez6kGb0YDcR9DSqaujR4y4vydZu1jqV+8B9 ZT8hP++SrGWf5IFaiBMaNVLQUI7mPNpT4pgsrnPjVp/9umYx2IGdMd5azUTuJct/ smWnXgegDVS8FUWtLqcwnF2rHFGtzHrg/62TofVeVi2S9VMWhHSf36B2qIzptdYC sWEInZWt87dQHg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=ejEInF/oIZxv7U7gX9AlaXmjytYaU dccqrz3Z+CKqTk=; b=PznuiJDBajktn0Svj1e/1QN/DqOLH8OZjlwODmcjvBsqL B2f0GXu8QeoGfn3qsgR62O8lGNicZdnCJkoWTQAy4Eja15wAARpS9pvHFd9L4iE7 zNEAQ4PbrQTtmGW231TIuCDkJWAIi7nhrxg847JMM/74d0w4MA6xwqWu4Ce94I/C 8hOneFmWOl2H5iV9H3frhUu5Vvn/epbwe8XT69enwFWTdtIn0wvjsWfHbmyvy7j1 nFywpI9x8kDHgHzNfy+uDzAn0cpNK5xDRVtYWsH3QWsk86TTp3BwBBeQ7JLveK5J mD75sx0IHWMXTNH160eHDJtylqo2WW3pYTgrbK7FQ==
X-ME-Sender: <xms:zf8OWUivD7uf5ko0cXng8YnhimrjB19StYT6T-n3hiiUtjzQ-4HHKQ>
Received: by mailuser.nyi.internal (Postfix, from userid 99) id A1AC9BAB6F; Sun, 7 May 2017 07:06:53 -0400 (EDT)
Message-Id: <1494155213.3081934.968406608.5F9E8767@webmail.messagingengine.com>
From: Bron Gondwana <brong@fastmail.fm>
To: jmap@ietf.org
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"
X-Mailer: MessagingEngine.com Webmail Interface - ajax-6cc55fe1
Date: Sun, 07 May 2017 21:06:53 +1000
Archived-At: <https://mailarchive.ietf.org/arch/msg/jmap/7dSQsqRBJ_YlZ7wF83i2ihi14XI>
Subject: [Jmap] Hacker News Discussion - getMailboxes single data set vs tree primitives or paging
X-BeenThere: jmap@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: JSON Message Access Protocol <jmap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/jmap>, <mailto:jmap-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/jmap/>
List-Post: <mailto:jmap@ietf.org>
List-Help: <mailto:jmap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/jmap>, <mailto:jmap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 07 May 2017 11:06:56 -0000

https://news.ycombinator.com/item?id=14284526

It's a rather silly discussion that went off onto a tangent, that's HN for you.  The key interesting point is this:

A quick look at the protocol looks like in order to get a list of mailboxes, you have to ask for them all. In IMAP, you can say, show me top level mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous role, I was responsible for working on a mail archiving solution which had a large hierarchy of folders (into the thousands IIRC) in a single account. Access to the archive was via IMAP. Looks to me that JMAP here would require the client to potentially fetch megabytes of data just to start showing a list of folders, where the solution we had with IMAP probably took a couple of kilobytes at most to get started.

...

and the followup:

> Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too.

Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs.

...

I think it is worth discussing the tradeoffs in terms of additional complexity to offer paging, tree filtering (parentIds: [...]) or both in the standard.  The pro side, you can access potentially millions of mailboxes without needing to know all the folder ids up front.  On the con side, it's more complexity for the server (client can always just fetch everything anyway if it doesn't care) - and most clients seem to fetch the entire tree anyway, so it just means more round trips for them.

Bron.

-- 
  Bron Gondwana
  brong@fastmail.fm