Re: [Jmap] Hacker News Discussion - getMailboxes single data set vs tree primitives or paging

[Mike Cardwell <[e-mail]>]

> https://news.ycombinator.com/item?id=14284526
> 
> It's a rather silly discussion that went off onto a tangent, that's HN for
> you.

I am the person who is responsible for that silly discussion and also ironically
the person who said he wasn't invested in the success of JMAP enough to bring it
to this very list. Well, here I am.

> Me:
>  A quick look at the protocol looks like in order to get a list of mailboxes,
>  you have to ask for them all. In IMAP, you can say, show me top level
>  mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous
>  role, I was responsible for working on a mail archiving solution which had a
>  large hierarchy of folders (into the thousands IIRC) in a single account.
>  Access to the archive was via IMAP. Looks to me that JMAP here would require
>  the client to potentially fetch megabytes of data just to start showing a
>  list of folders, where the solution we had with IMAP probably took a couple
>  of kilobytes at most to get started.

> and the followup:

> Bron:
>  Of course nothing stops you storing thousands of mailboxes in a
>  non-heirarchical layout and making IMAP have large LIST fetches too.

> Me:
>  Yes. I would issue the LIST command, and as the IMAP server started
>  streaming the results to the client, I would immediately process each mailbox
>  as soon as it was down, rather than waiting for them all to be retrieved. I
>  fear that with a JSON based protocol, most server and client implementations
>  wouldn't deal with streaming (although they technically could), and would just
>  deal with completed blobs.

> I think it is worth discussing the tradeoffs in terms of additional complexity
> to offer paging, tree filtering (parentIds: [...]) or both in the standard.
> The pro side, you can access potentially millions of mailboxes without needing
> to know all the folder ids up front.  On the con side, it's more complexity for
> the server (client can always just fetch everything anyway if it doesn't care)
> - and most clients seem to fetch the entire tree anyway, so it just means more
> round trips for them.

I know it's from a different time, but in the RFC for IMAP implementation
recommendations, it says the following:

https://tools.ietf.org/html/rfc2683

   Some servers present Usenet newsgroups to IMAP users.  Newsgroups,
   and other such hierarchical mailbox structures, can be very numerous
   but may have only a few entries at the top level of hierarchy.  Also,
   some servers are built against mail stores that can, unbeknownst to
   the server, have circular hierarchies - that is, it's possible for
   "a/b/c/d" to resolve to the same file structure as "a", which would
   then mean that "a/b/c/d/b" is the same as "a/b", and the hierarchy
   will never end.  The LIST response in this case will be unlimited.

   Clients that will have trouble with this are those that use

       C: 001 LIST "" *

   to determine the mailbox list.  Because of this, clients should not
   use an unqualified "*" that way in the LIST command.  A safer
   approach is to list each level of hierarchy individually, allowing
   the user to traverse the tree one limb at a time

IMAP can essentially work with folders of infinite depth when clients
don't request the full list of folders up front. It can also work
with slow backends. What if you had a giant mail store backed by
Amazon S3? Your data could be spread over multiple buckets. Why wait
for all buckets to be retrieved before you start sending data to
the client? IMAP also lets you present a dynamically generated folder
structure where each new level is calculated on demand. Perhaps
someone will want to stick a dynamic JMAP frontend in front of the
https://www.mail-archive.com/ website where the folder structure is
"List Name/Post Year/Post month". That would be a lot of folders.

I posit that allowing you to request a complete list of all folders in
one go was a mistake in IMAP, and one that shouldn't be repeated in
JMAP.

-- 
Mike Cardwell  https://www.grepular.com
OpenPGP Key    DF70 D8E5 FBD6 8519 9257  C44C 0DA6 8B1E 1801 A332
XMPP OTR Key   8924 B06A 7917 AAF3 DBB1  BF1B 295C 3C78 3EF1 46B4