Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Ted Lemon <[e-mail]>]

Thursday, Dec 3, 2015 1:41 PM Jakub Moscicki wrote:
> Could you please explain why you think it does not scale? There are schemes in which you do not need to propfind the entire remote tree to discover changes, only a sequence of propfinds in a direct path leading to a changed entity. That’s not optimal for sure (several calls needed, depending on the depth of the change) but if it comes to WebDAV itself, apart from a bloated XML representation, I do not see much problem.

It doesn't scale because you've just described a lockstep process for locating changes; instead of there being one transaction to get the set of changes from a known common version, you have multiple steps, and the number of steps increases at least logarithmically (I am guessing, since I don't know your algorithm) as the number of changes since last sync increases or the size of the tree increases.   If you are syncing continuously this may not be a big deal, but I don't think that's a reasonable assumption.

This also has the problem that you have no separation of metadata and data, and no versioning, which means that you have to have a client-server model, and you can't have a distributed peer model.

> Okey, but what you need to remember the client state and pass it on to the server to calculate the diff state. Any good ideas how to do that?

Keep the metadata sorted (e.g. by modification time), and do a diff whenever a new version is generated.   Versions that have never been shared with other peers should be trimmed once they are not the head version.   When peer A wants to sync with peer B, it announces the version it has and the version it last remembers syncing with peer B.   If Peer B remembers that version, then the difference can be computed trivially, O(n), and is minimal (n*2 at worst), where n is the number of changes.   If Peer B has lost its memory, then it has to send the complete inventory of its head version, which Peer A can then compare with its head to see what's changed.


--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[e-mail]