[Jmap] recommendations for gigantic mailboxes

[Eric Wong <[e-mail]>]

Hi all, I'm considering implementing JMAP in public-inbox[1].

TL;DR: would it be reasonable to expose 10 million messages in a
single JMAP mailbox?  It can already be done for an NNTP newsgroup :>


public-inbox started with NNTP and HTTP support, but I recently
started implementing an IMAP server and it seems fine.  It's
mainly tested with mutt, isync|mbsync, offlineimap, and Gnus.

Whereas newsreaders typically have a rolling window where
messages expire; MUAs and IMAP <=> Maildir syncing tools tend to
hold messages forever.  That's slow when mailboxes get into the
order of 100K...

MUAs and Maildirs on traditional filesystems fall down with 400K
messages in the [e-mail] mailing list, so right now
it's broken into 9 segments capped at 50K messages, each:

  imaps://public-inbox.org/INBOX.comp.version-control.git.0
  ..
  imaps://public-inbox.org/INBOX.comp.version-control.git.8

400K is nothing, even.  LKML has 3.6 million and grows around
30K a month[2]:

  https://lore.kernel.org/lkml/
  nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel

So, JMAP is fresh ground; so I wonder if having a single mailbox
on the order of 10 million messages is reasonable for JMAP clients.
NNTP clients have no problem handling a newsgroup that large.

Or if I should go route I did with IMAP and partition mailboxes
virtually into 50K or so.  It doesn't affect the underlying
storage at all.  I wasn't really happy to do that for IMAP, but
the prevalance and limitations of existing MUAs had to be taken
into account (and I still work on a Centrino laptop from 2005,
sometimes).

Thanks for reading!


[1] https://public-inbox.org/README.html
    Archives mail in git; indexes using Xapian + SQLite.

[2] LKML required designing the v2 public-inbox format
    back in 2018 for scalability:
    https://public-inbox.org/public-inbox-v2-format.html
    I've tested it up to 10M or so before running out of space;
    but I expect it to do more in ~10 years (if civilization
    survives that long!)