Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Hugo González Labrador <[e-mail]>]

On Wed, Dec 2, 2015, at 01:30 PM, Michiel de Jong wrote:
> Hi all! Thanks for all your reactions to the remoteStorage Internet-
> Draft.
>
> We get the question about WebDAV a lot, in the next version we
> should add a remark about it in the intro. The folder descriptions
> returned when you GET a URL that ends with a / are indeed a
> deviation from the XML returned by WebDAV server, and this is just
> because nowadays JSON is easier to use than XML for developers, both
> client-side and server-side.

I totally agree here, this was going to happen soon or later and it is
really appreciated.

> The fact that we don't require servers to support WebDAV's custom
> verbs like PROPPATCH etc. is for three reasons:
> 1) it's a lot of work to implement this without using an existing
>    WebDAV library
> 2) in practice, a lot of WebDAV servers get it wrong, or don't
>    implement all of WebDAV. It's very annoying for client implementers
>    to have to deal with servers that e.g. chose not to implement LOCK
>    and UNLOCK.
> 3) we don't really need all these advanced features on top of standard
>    HTTP, just supporting GET/PUT/DELETE for resources, and adding a
>    simple folder description format, is enough for most applications.
>
> Other than that, the remoteStorage Internet-Draft specifies a *lot*
> more than just how each HTTP verb should behave:
> * requiring support for OAuth implicit-grant flow
> * requiring ETag support and nested versioning (i.e. the folder's ETag
>   changes if anything within that folder changes)
> * requiring CORS headers
> * requiring a WebFinger announcement for service discovery It would be
>   easy to add these three things on top of WebDAV, instead of putting
>   them on top of our minimal GET/PUT/DELETE API definition. In fact,
>   we could probably separate it into two Internet-Drafts: one for the
>   'RESTful folders' API which is our simplification of WebDAV, and a
>   second one for OAuth/ETags/CORS/WebFinger on top of either WebDAV or
>   'RESTful folders' or whatever other HTTP-based API you want.

There is one requirement that all synchronisers need to handle: moving
resources. In this spec the alternative of a WebDAV MOVE is not
specified.

It is true that a MOVE could be replaced with a DELETE + UPLOAD but that
is not acceptable in most cases due to the inefficiency of such
operation (cpu, bandwidth consumption ...)

Is there a plan to support such basic feature?

>From the current remoteStorage spec, the ownCloud sync protocol can be
implemented. The missing bit is tracking those remote moves.

Cheers

> On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador
> <[e-mail]> wrote:
>> __
>>
>>
>>
>>
>> On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
>>> Hi
>>>
>>> On 周三, 12月 2, 2015 at 17:56,  Hugo González Labrador
>>> <[e-mail]> wrote:
>>>>
>>>>
>>>>
>>>> On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
>>>>> Hi
>>>>>
>>>>> 2015-12-02 5:14 GMT+08:00 Hugo González Labrador
>>>>> <[e-mail]>:
>>>>>> Hi,
>>>>>>
>>>>>> from my point of view the remoteStorage project addresses a
>>>>>> subset of the use cases of the  WebDAV specification.
>>>>>>
>>>>>> The main difference I can observe is that the specification is
>>>>>> built on REST instead of XML-based communication.
>>>>>>
>>>>>> I personally like much more REST than WebDAV because it is more
>>>>>> developer friendly and it is faster to develop. Maybe the
>>>>>> remoteStorage API becomes the next WebDAV :)
>>>>>>
>>>>>> However, the remoteStorage API does not provide a way of
>>>>>> synchronising data, this task is delegated to the developers. Is
>>>>>> there a plan to provide such feature based on the outcome of this
>>>>>> draft?
>>>>> I'm a little bit confused why you say the remoteStorage does not
>>>>> provide that. Is that because remoteStorage does not perform like
>>>>> a typical sync services (e.g. dropbox...) or you are saying
>>>>> something else?
>>>> Yes, because it does not offer synchronisation capabilities.
>>> Got it. And What I am wondering is that do we need to include those
>>> capabilities in a base specifications. Since it is hard to
>>> standardize the capabilities like dedup or delta. Maybe a better way
>>> is to define those in a separate specification,
>>
>
> Thanks for giving these examples - so by
'synchronisation capabilities' you mean 1) deduplication and 2) delta
updates? Anything else or is that an exhaustive list?
>>
>>> something like extensions. For a base document, we just need to
>>> define how to perform sync operations.
>>
>>
>> Yes, I agree that may be  an extension of this draft could handle the
>> synchronisation part.
>>
>>
>>
>
> Our Internet-Draft is heavily focused on the world wide web, whose
> URLs are not content-addressable, we can't change that. So that
> architecture is not very friendly to deduplication, compared to for
> instance BitTorrent. As you already said, developers can still dedupe
> on the server-side when storing blobs to disk, and can also dedupe on
> the client side before creating the resources the client uploads. As
> far as I know, delta updates are not supported by the ETag system -
> you cannot do a range request to find out if certain bytes within a
> document have changed. However, the folder system we define does
> encourage you, for instance when you develop a To Do List app, put
> each task on its own document, and then query the folder to see which
> of them changed, instead of putting them all in one big document and
> retrieving the whole document each time.
>
> Cheers, Michiel.
>
>>
>>>>
>>>>>>
>>>>>> BTW, I want to introduce ClawIO (http://clawio.github.io), a
>>>>>> research project to benchmark different synchronisation protocols
>>>>>> against different data backends with special attention to provide
>>>>>> a common sync API.
>>>>>>
>>>>>> A common API for different sync protocols is being created based
>>>>>> on the architecture specified in this draft (control and data
>>>>>> servers) and the
>>>>> I cannot find a distributed architecture in this draft. It seems
>>>>> that they handle metadata and content data together, just like a
>>>>> normal web service.
>>>>
>>>> ClawIO is fully distributed. Every logical unit is a different
>>>> server than be scaled out. Data and Metadata channels are
>>>> independent from each other and reside on different servers.
>>> That is widely employed in popular sync services. And that is also
>>> beneficial to privacy to some extent. But in the context of sync in
>>> browser (which is mainly considered in the remoteStorage), I don't
>>> know whether this is reasonable. But I think we should deploy
>>> distributed architecture although it will make things complicated.
>>
>>
>> Of course, the remoteStorage is targeted to browsers, so syncing does
>> not make too much sense in this case. With the rise of Linux
>> container micro-service based architectures, the deployment of  such
>> highly complex systems should become easier and faster.
>>
>> Best,
>>
>>
>> Hugo
>>
>>
>>> Regards, Linhui
>>>>
>>>>
>>>>> Regards, Linhui
>>>>>> one from the CERN EOS project (management, disk and queue
>>>>>> servers).
>>>>>>
>>>>>> The Phase I has implemented the ownCloud Sync Protocol and Phase
>>>>>> II will implement the SeaFile Sync Protocol. The choice of these
>>>>>> protocols among others is because they are really opposed to each
>>>>>> other in terms of syncing (delta vs non-delta, state-based vs log/event/git-
>>>>>> based sync …), so finding a common approach is more challenging.
>>>>>>
>>>>>> Providing a base specification/architecture to measure the
>>>>>> feasibility of this draft is one of the objectives of the
>>>>>> project.
>>>>>>
>>>>>> I believe that the work being done here and in ClawIO are
>>>>>> supplementary to each other and I think mutual collaboration
>>>>>> could be beneficial for both sides.
>>>>>>
>>>>>> Also, if there is interest, the remoteStorage API can be added to
>>>>>> ClawIO.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Hugo Gonzalez Labrador
>>>>>>
>>>>>> On Tue, Dec 1, 2015, at 09:00 PM, [e-mail]
>>>>>> wrote:
>>>>>> > Send Storagesync mailing list submissions to
>>>>>> > [e-mail]
>>>>>> >
>>>>>> > To subscribe or unsubscribe via the World Wide Web, visit
>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync or, via
>>>>>> > email, send a message with subject or body 'help' to       storagesync-
>>>>>> > [e-mail]
>>>>>> >
>>>>>> > You can reach the person managing the list at       storagesync-
>>>>>> > [e-mail]
>>>>>> >
>>>>>> > When replying, please edit your Subject line so it is more
>>>>>> > specific than "Re: Contents of Storagesync digest..." Today's
>>>>>> > Topics:
>>>>>> >
>>>>>> >1. New version of draft-dejong-remotestorage    Internet-Draft
>>>>>> >available (Michiel de Jong)    2. Re: New version of draft-dejong-
>>>>>> >remotestorage Internet-Draft       available (Gihan Dias)    3.
>>>>>> >Re: New version of draft-dejong-remotestorage Internet-Draft
>>>>>> >available (Fei Song)
>>>>>> > _______________________________________________
>>>>>> > Storagesync mailing list [e-mail]
>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync Email had 3
>>>>>> > attachments:
>>>>>> > + [Storagesync] New version of draft-dejong-remotestorage Internet-
>>>>>> >   Draft available   2k (message/rfc822)
>>>>>> > + Re: [Storagesync] New version of draft-dejong-remotestorage
>>>>>> >   Internet-Draft available   1k (message/rfc822)
>>>>>> > + Re: [Storagesync] New version of draft-dejong-remotestorage
>>>>>> >   Internet-Draft available   2k (message/rfc822)
>>>>>>
>>>>>> _______________________________________________
>>>>>> Storagesync mailing list [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/storagesync
>>>>
>>