Re: [Tools-discuss] matrix tests

[Matthew Hodgson <[e-mail]>]

Better late than never I can try to provide more clarity on Matrix's 
data processing/retention semantics - please see comments below...

On 13/12/2020 23:02, Glen wrote:

> Stephen Farrell <[e-mail]> wrote:
>> It didn't occur to me to create any matrix rooms. If it had, I'm not
>> sure if that'd have been useful but I can imagine it being so in
>> future, e.g. for a bar bof. Before that'd be a standard procedure I'd
>> like to understand the duration for which any such rooms might last,
>> how closed/open they may be etc.
>
> I can actually report some technical data on that.
>
> As I was performing server maintenance over the past few days, I 
> discovered that the Matrix database was taking up a significant amount 
> of space (91 GB for 37 local users.).

There are several reasons for this:

  * The server has been provisioned using sqlite rather than postgres as 
the DB.  Sqlite is meant to be just for evaluation, not for production, 
and I dread to think how fragmented or unoptimised a 91GB sqlite DB has 
got :)
  * Synapse's schema is currently quite naive on how it snapshots the 
state of its chatrooms.  It currently takes a snapshot every 100 state 
changes (where state changes are things like room membership changes; 
not normal messages).  Thus a room with 10,000 members causes another 
10,000 db rows to be inserted every 100 joins or parts.  This is 
obviously terrible; there's a workaround to apply a better compression 
algorithm via https://github.com/matrix-org/rust-synapse-compress-state, 
which needs to be merged into Synapse proper.
  * As you've pointed out, Matrix works by replicating conversations 
between servers.  So even if you have 1 user, if they go nuts and try to 
join 1,000 busy rooms with huge numbers of participants, they can suck 
up a lot of disk space.  Conversely, if you have 1,000 users and they 
are all in tiny rooms, the opposite could be true.  Matrix is more like 
hosting an IMAP server than (say) an SMTP or XMPP server in this respective.
  * You can configure the server to have limited the data it stores for 
the rooms its users are in, and I assume this hasn't been configured: 
https://github.com/matrix-org/synapse/blob/44b7d4c6d6d5e8d78bd0154b407defea4a35aebd/docs/sample_config.yaml#L410-L421.

> I put in an inquiry but didn't find anyone online (likely people are 
> offline for the weekend) so I did some digging myself.

Sorry - I think our messages crossed while you were writing the original 
mail :)

> Unlike Jabber, where a room is hosted on a single server; in Matrix, 
> it turns out that every room that anyone belongs to is co-hosted on 
> every server used by any of that room's users. [1] Even if all (local) 
> users leave the room, the room remains on the server, along with all 
> of the shared data.

This isn't strictly true.  If all local users leave the room, and 
"forget" it (equivalent to expunging an IMAP folder - it's an option 
after you leave a room in the 'Historical' section in Element), then the 
data is free to be garbage collected.

> Moreover, if I understand what the developers explained to me, it is 
> not possible to delete a room "everywhere".  I can delete a room that 
> exists on the local server, but that does not delete the room from 
> other servers to which the room was shared.

In theory you could request other servers delete the room and hope they 
honour the request.  However, i'm not sure that I'd like some random 
admin/moderator/user on a remote server to decide to reach in and 
destroy my copy of a room on my server - so we haven't implemented it yet.

> And if our server somehow became disconnected from the rest, all of 
> the rooms that had been shared out to other servers would continue to 
> exist on those other servers forever as orphaned clones (for, as we 
> like to say, some value of "forever"), outside of our ability to 
> control (or perhaps even enumerate.)

They wouldn't really be "orphaned clones", any more than a local git 
repository clone is an "orphaned clone" of whatever you cloned it from.  
To be orphaned implies you once had parents.  In Matrix, the rooms are 
shared over their participating servers, and all servers are equal in 
that respect.

> So unlike operating a Jabber server, where the conference server is 
> the "authoritative" location of the room (and logs are captured 
> there), and, it turns out, unlike operating a Zulip server (which also 
> has an authoritative location).... operating a Matrix server isn't 
> really "operating" a server so much as "joining a huge network of peer 
> servers."

Yup, precisely. It's the same difference between using SVN and using 
Git.  Most people consider the lack of authoritive server to be an 
extremely desirable feature - it means that if I join IETF rooms from my 
personal server, I have full sovereignty over my copy of the 
conversation, even if my 'net connection goes down, or even if something 
terrible happened to the IETF server, etc.  I have network partition 
tolerance.

> As the IETF's operator, I do not set policy, so this is just my 
> opinion... but I *think* that having our Matrix server mirroring some 
> of the rooms in my list above seems... problematic. 

I would expect this to be no worse than (say) running a caching HTTP 
proxy for users.  If your users choose to go view weird stuff out there 
on Matrix, then your server ends up with a local cache of that data 
until it expires.

However, if you wanted to limit the servers your server can federate 
with, that's possible too (but not very idiomatic - just as it wouldn't 
be idiomatic to specify a whitelist or blocklist of email servers to 
talk SMTP with): 
https://github.com/matrix-org/synapse/blob/44b7d4c6d6d5e8d78bd0154b407defea4a35aebd/docs/sample_config.yaml#L673-L695

> If it turns out that this is the behavior desired by the IETF, then no 
> changes are needed. If, however, the IETF wants to operate the Matrix 
> server in a more-or-less authoritative, "Note Well"-compliant kind of 
> way, then I *think* the Matrix server would need to be deployed with 
> federation disabled (meaning, local user signups only, and no room 
> sharing across the world), and public room sharing disabled as well.  
> We can certainly do this, but we have not tested that kind of setup yet.

You could do this, but it would very much miss the fundamental nature of 
Matrix - the equivalent of realising that because an internet-connected 
LAN could be used by its users to access unexpected content on the 
internet, one should isolate it from the internet.  I think this might 
be counter to the IETF's aethos :)

In terms of being authoritative in a Note Well kind-of way, it's worth 
noting that the replicated copies of rooms which other servers maintain 
are cryptographically signed to match each other.  So from a logical 
perspective, they do not diverge (other than during a netsplit).

> But unless they make that choice, then the answer to Stephen's 
> questions seem to be:
>
> > the duration for which any such rooms might last,
>
> "forever", with no option to delete/close unless federation is 
> disabled, and

This is not correct. The server admin can boot rooms off the server, or 
specify a retention policy, or the room admin can specify a retention 
policy.  If all users have left and forgotten the room, the server could 
garbage collect it (if desired, although this hasn't landed in Synapse 
yet). Federation has no impact on the ability to delete/close a room.

 > how closed/open they may be etc.

> "wide open to every Matrix server of every participating user."

This is not necessarily true: if data privacy is a concern, you can 
enable end-to-end encryption on the room, and then the data is not 
visible to *any* server - just the authorised participating users' devices.

Aside from end-to-end encryption, Matrix rooms also impose decentralised 
ACLs to limit which users and servers can see the history based on the 
room admin's requirements; it turns out this is a novel area of computer 
science: 
https://matrix.org/blog/2020/06/16/matrix-decomposition-an-independent-academic-analysis-of-matrix-state-resolution.

So while it's true that the new messages in an unencrypted room is wide 
open and visible to all servers which have users currently participating 
in that room, it is not true to say that past history is visible (unless 
explicitly shared), and future messages are not replicated after a 
server's users leave.

Hope this provides some clarity.  Apologies that the mental process of 
familiarisation with Matrix is a paradigm shift similar to going from a 
VCS to a DVCS, but when it clicks, much like a DVCS, its characteristics 
hopefully feel desirable rather than unexpected and weird.

thanks,

Matthew


-- 
Matthew Hodgson
Matrix.org