Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Ted Lemon <[e-mail]>]

BTW, it occurs to me that at CERN you deal a lot in very large data sets which are created once and never modified, because they are records of physical events.  So the idea of merging might not seem all that important, because you will never do it on these data sets.

However, one thing that a good versioning system allows for that I think you should consider important is the ability to avoid accidentally losing data.   When you have really big data archives, one of the things that you want to do for redundancy is keep multiple copies.   If one instance gets corrupted, you want to be able to detect that, and you want to be able to avoid losing other instances of the file if the file is lost from an instance of a folder.   The most reliable way to avoid that is to keep versioning metadata.   Multiple copies of versioning metadata are very useful for forensic analysis when something goes wrong, as well.

Additionally, while the bulk of the actual _data_ that CERN stores is really big files, the little files matter just as much--the work researchers are doing, particularly collaboratively.   Enabling efficient collaboration on articles in progress, enabling effective sharing of code, and so on, all are very important despite representing a tiny percentage of the total data stored.


--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[e-mail]