[Storagesync] recent issues discussed (plain text)

["Fei Song" <[e-mail]>]

Here is the latest version. Please email me if anything is missed:

1.The design targets of WebDAV, rsync and other existing approaches?
2.The potential use cases of ISS, such as client/server, git-like pattern, svn, etc.
3.The efficiency improvements might be the second goal for standardizing ISS protocol
4.CORS headers on storage sync APIs
5.What is needed for the ISS: a sync protocol or a generalized API
6.remoteStorage draft discussion
  a)relationship vs WebDAV
  b)MOVE action (synchronization) should be added or not
  c)Beside web browser, desktop apps (by hacking way)
  d)comics of new standard
  e)etag issues vs metadata
    i.is mainly for identifying whether a document is changed or not
    ii.is easy to implement than that of WebDAV sync protocol or not
    iii.the metadata file contains all etags for all files at both client and server side or not
  f)the distributed peer model (no server) and C/S mode
  g)a fancy example (with pics) of OfflineIMAP’s sync process in following URL
    http://blog.ezyang.com/2012/08/how-offlineimap-works/
7.GitHub (instead of email messages) has been created:
  https://github.com/labkode/Internet-Storage-Sync
  a)What is the topic? 
    i.Whether it is suitable to build on WebDAV
    ii.WebDAV vs remoteStorage
    iii.Advantages vs disadvantages of WebDAV
8.Metadata and data separation scheme and platform for synchronization
  a)ownCloud when configured to use an Object Storage as the Primary User Storage. (Metadata is handled by a MySQL database)
  b)CERN EOS Storage System. (Metadata is handled in high performance in-memory structures).
  c)DropBox. (As far as I know it uses S3 behind. For metadata it is unknown, but probably not on S3). 
    Paper for analyzing DropBox:
    http://annasperotto.org/papers/2012/imc140-drago.pdf
  d)ClawIO will have an implementation of this approach in the next phase using Swift.
9.Whether we should keep metadata history or modification history or action history at server side (or other places)
10.How to handle the “conflict discovery (resolution)” issues
  a)A good demonstration of this issue was given in details (File F, Client A, Client B, etc.)
  b)Should it be a layer above syncing?
  c)Should it depend on the use case?
  d)Should it be a one-size-fits-all approach?
  e)How many different kinds of conflict?
    i.File level conflicts, e.g. both remote (server) and local (client) have changed since last sync.
    ii.Interleaved level conflicts, e.g. one client makes a change based on version X of the file, a second makes a change based on version X+1, the second one commits, and then the first one commits.
    iii.…
  f)The sequence (order) of changes in one file is important or not (quite similar with “item 9” in this list), perhaps depends on the file type (or use cases):
    i.It may be important for: sound file, drawing file, latex source files of articles, iWork files, Office files, images, pdfs, source code, etc.
    ii.It may be not important for: Physics data.
  g)Whether to enable an automatic merging might be a research problem or a further use case?
  h)If It is so hard to do conflict resolution even for simple and well-structured “vcard case”, should we handle this issue by separating files based on their property?
11.A de-facto sync protocol description used by ownCloud sync clients (and some footnote discussion on future development directions). This document comes with a test suite (partially implemented) that verifies if a server adheres to this specification. The URL is as follow:
  https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md


Some related organizations, events, projects, etc.: 
GEANT community, OpenCloudMesh, ownCloud, CS3, remoteStorage, ClawIO, crosscloud, Dropbox, CERN EOS Storage System, to be added…