Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Jakub Moscicki <[e-mail]>]

Hello,

Last time I looked at this in cernbox we had more than 1000 file extensions and quite a large fraction of “typical dropbox” types of files (documents of different sorts, latex source files of articles, iWork files, Office files, images, pdfs, source code (!), etc). We also have an increasing number of physics datasets and we will have many more as we provide direct connection via sync/share to the already existing huge collections of these datasets. For the data files at CERN, yes you are right, there is no point of merging them and they quite often are never modified. But I would also tend to disagree with your statement that most of files are text files — as was explained by Marc Blanchet — in general it is not the case, not only for CERN.

Best regards,

kuba

--

> On 09 Dec 2015, at 15:43, Ted Lemon <[e-mail]> wrote:
> 
> BTW, it occurs to me that at CERN you deal a lot in very large data sets which are created once and never modified, because they are records of physical events.  So the idea of merging might not seem all that important, because you will never do it on these data sets.
> 
> However, one thing that a good versioning system allows for that I think you should consider important is the ability to avoid accidentally losing data.   When you have really big data archives, one of the things that you want to do for redundancy is keep multiple copies.   If one instance gets corrupted, you want to be able to detect that, and you want to be able to avoid losing other instances of the file if the file is lost from an instance of a folder.   The most reliable way to avoid that is to keep versioning metadata.   Multiple copies of versioning metadata are very useful for forensic analysis when something goes wrong, as well.
> 
> Additionally, while the bulk of the actual _data_ that CERN stores is really big files, the little files matter just as much--the work researchers are doing, particularly collaboratively.   Enabling efficient collaboration on articles in progress, enabling effective sharing of code, and so on, all are very important despite representing a tiny percentage of the total data stored.
> 
> 
> --
> Sent from Whiteout Mail - https://whiteout.io
> 
> My PGP key: https://keys.whiteout.io/[e-mail]_______________________________________________
> Storagesync mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/storagesync