Re: [Storagesync] Some preliminary investigations on ownCloud

[Linhui Sun <[e-mail]>]

2015-11-26 15:29 GMT+08:00 Jakub Moscicki <[e-mail]>:

> 2015-11-26 11:28 GMT+08:00 Fei Song <[e-mail]>:
>
>> BTW, Based on the last sentence of last email:"The intention of this
>> message is to investigate the current state of using WebDAV for sync
>> purposes to see what needs to be improved here and whether we need new
>> protocols"
>>
>> The outcome he/she wanted might be just the links like
>> http://cs3.ethz.ch/program.html :)
>>
> In another word, I think what I want finally might be a discussion about
> "What is needed for the ISS: a sync protocol or a generalized API". Sorry
> for the poor expression : )
>
>
> I think this is a relevant question indeed (well, actually also if there
> is any standard needed at all in the first place). By the generalized API
> do you mean a HTTP-style API (like REST)? In that case you may consider
> WebDAV quite close and the difference between the protocol and the API
> blurs for practical purposes.
>
Yes. And maybe for the application layer applications, an API is more
realistic and practical at the first stage? I'm not saying we should not
standardize a sync protocol...

>
> While I agree that WebDAV may have disadvantages there is a good number of
> installations using this already in our community (research labs mainly in,
> but not limited to, Europe). I think the interesting point about
> WebDAV/HTTP is extensibility and maybe it is worth is that these extensions
> go by a “standard”. However, you should consider that for
>
Actually I don't know what the "extensibility" means here, are you saying
that we could extend the WebDAV?

> reasonably efficient sync scenario the server should also exhibit certain
> behaviour. That is, in case of owncloud for example, a server should be
> able to efficiently propagate the ETAG changes up the directory tree (like
> the Merkle Tree) so that the client may use propfind efficiently. This is
> not a hard spec requirement but otherwise propfinding the entire remote
> tree each time would be impractically inefficient. So this really goes a
> little bit beyond just an API.
>
And that is what I mean the "applicability", IMO, the WebDAV seems not
suitable for sync purposes. As a result, the developers compromise to
propfind the entire file tree at intervals.

>
> You should also consider that the OpenCloudMesh (under GEANT umbrella) is
> an initiative with the intent is to make cross-service sharing very easy.
> These shares may also be synchronized automatically. I currently have no
> evidence, however, that other software providers with the exception of
> owncloud are interested in developing such standard. If there is no bottom
> up interest from the users then it won’t happen (in my opinion).
>
That's cool. I'm just wondering whether the ISS could get some input from
that...

>
> With the link I wanted to point you to the fact that what you discuss in
> this mailing list will be also discussed at the upcoming CS3 event I linked
> in. The intent is to do this discussing together between service providers,
> developers and researchers together — so that it does not only end up as an
> academic exercise but backed up by a critical mass if there is some
> potential in standardisation, at least in our community. I hope this could
> be of interest to IETF community, as mentioned on the program page.
>
> Selected papers will be published in FGCS after the event, so please stand
> by, or attend the event if you want to be part of this discussion here.
> BTW. the abstract submission deadline is past but one exceptionally good
> contribution could still possibly be accommodated if submitted rapidly.
>
> I would be nonetheless happy to continue contributing to the discussion in
> this mailing list.
>
Thanks for that!

>
> Best regards,
>
> Jakub Moscicki
>
> --
>
>
>
> Regards,
> Linhui
>
>>
>>
>> --------------
>> Fei Song
>> >Hello,
>> >
>> >What kind of outcome are you looking for with this analysis? Some
>> research in this area has already been done or is being done as we speak
>> >
>> >e.g. "A study of delta-sync and other optimisation in HTTP/Webdav
>> synchonisation protocols"
>> >
>> >see "Technology and Research":
>> >
>> >http://cs3.ethz.ch/program.html
>> >
>> >It would be interesting to see if there is a potential for
>> collaboration. Or maybe we already have some information you are looking
>> for.
>> >
>> >Best regards,
>> >
>> >Jakub Moscicki
>> >
>> >—
>> >
>> >
>> >On 25 Nov 2015, at 11:45, Linhui Sun <[e-mail]<mailto:
>> [e-mail]>> wrote:
>> >
>> >Hi all,
>> >
>> >As I mentioned before, I think the developers could benefit from the
>> IETF standards. The ownCloud (https://owncloud.org/) is just an example.
>> It is developed for those who do not trust commercial storage services and
>> want to build their own network-based storage services. The ownCloud is
>> using WebDAV (RFC4918) to achieve the data sync. IMO, the WebDAV is
>> designed for distributed work but not for the sync. Thus, I made some
>> preliminary investigations on how the ownCloud uses WebDAV for sync
>> purposes. A brief summary of what I've found is in the following, please
>> correct me if I am wrong.
>> >
>> >I installed the ownCloud server (v8.2.1) on the CentOS7, and the client
>> is a desktop client on Windows.
>> >
>> >1. To find whether there is a change to the synced directory, the client
>> continuously sends PROPFIND to the server at regular intervals (around 34
>> seconds under my observation). The server will respond a 207 Multi-Status
>> Response to tell whether the main directory has been changed. To perform
>> this regular check, the client will open a new TCP connection to send the
>> PROPFIND, the server will close the existing TCP connection after
>> responding the 207 Multi-Status Response. For the next check, the client
>> will open another new TCP connection.
>> >
>> >2. Every time adding (or creating) a new file to the local folder, the
>> client will open a new TCP connection (if there is no connection existing)
>> to send the file asap. The client will first send several PROPFINDs to find
>> out which sub-directory has been changed. And then it sends the file using
>> PUT. The server will respond a 201 Created Response and then terminate the
>> connection. Currently, I haven't found any application layer chunking, all
>> the segmentation are performed by TCP.
>> >
>> >3. Every time I delete (or rename) a file locally, the client will also
>> open a new TCP connection to send several PROPFINDs to find out which file
>> has been removed (or renamed). Then it will send DELETE (or MOVE). The
>> server will respond a 204 No Content Response (or 201 Created Response) and
>> then terminate the connection.
>> >
>> >4. I open a file and frequently edit and save it (actually this is what
>> I usually do with the Dropbox). The client will send the whole file to the
>> server every time I save the file.
>> >
>> >To summarize, it seems that the ownCloud makes heavily use of PROPFIND
>> to achieve the sync process. Each sync operation (e.g. upload, modify and
>> etc.) will start with sending one or more PROPFINDs. And currently, if I
>> add a file to the server (directly from the server side via web interface),
>> the client cannot find the change. I need to interrupt the sync and recover
>> it to make the client be aware of the change and download the newly added
>> file. I'm not sure whether this is caused by the sync mechanism or an
>> improper server configuration. I need to investigate this further and also
>> how the ownCloud works for multiple clients (or devices).
>> >
>> >For ISS, I think ownCloud has demonstrated to some extent that similar
>> IETF protocols could be deployed and employed. The intention of this
>> message is to investigate the current state of using WebDAV for sync
>> purposes to see what needs to be improved here and whether we need new
>> protocols.
>> >
>> >Comments are welcome : )
>> >
>> >Regards,
>> >Linhui
>> >
>> >
>> >_______________________________________________
>> >Storagesync mailing list
>> >[e-mail]<mailto:[e-mail]>
>> >https://www.ietf.org/mailman/listinfo/storagesync
>> >
>> _______________________________________________
>> Storagesync mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/storagesync
>>
>
>
>