Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Hugo González Labrador <[e-mail]>]

I propose to come up with a list of advantages and disadvantages of
using WebDAV and compare them against a JSON/REST based approach, like
remoteStorage.

Does it sound good ?


On Wed, Dec 2, 2015, at 01:59 PM, Linhui Sun wrote:
>
>
> 2015-12-02 20:43 GMT+08:00 Hugo González Labrador
> <[e-mail]>:
>> __
>>
>>
>>
>>
>> On Wed, Dec 2, 2015, at 01:30 PM, Michiel de Jong wrote:
>>> Hi all! Thanks for all your reactions to the remoteStorage Internet-
>>> Draft.
>>>
>>> We get the question about WebDAV a lot, in the next version we
>>> should add a remark about it in the intro. The folder descriptions
>>> returned when you GET a URL that ends with a / are indeed a
>>> deviation from the XML returned by WebDAV server, and this is just
>>> because nowadays JSON is easier to use than XML for developers, both
>>> client-side and server-side.
>>
>>
>> I totally agree here, this was going to happen soon or later and it
>> is really appreciated.
>>
>>
>>> The fact that we don't require servers to support WebDAV's custom
>>> verbs like PROPPATCH etc. is for three reasons:
>>> 1) it's a lot of work to implement this without using an existing
>>>    WebDAV library
>>> 2) in practice, a lot of WebDAV servers get it wrong, or don't
>>>    implement all of WebDAV. It's very annoying for client
>>>    implementers to have to deal with servers that e.g. chose not to
>>>    implement LOCK and UNLOCK.
>>> 3) we don't really need all these advanced features on top of
>>>    standard HTTP, just supporting GET/PUT/DELETE for resources, and
>>>    adding a simple folder description format, is enough for most
>>>    applications.
>>>
>>> Other than that, the remoteStorage Internet-Draft specifies a *lot*
>>> more than just how each HTTP verb should behave:
>>> * requiring support for OAuth implicit-grant flow
>>> * requiring ETag support and nested versioning (i.e. the folder's
>>>   ETag changes if anything within that folder changes)
>>> * requiring CORS headers
>>> * requiring a WebFinger announcement for service discovery It would
>>>   be easy to add these three things on top of WebDAV, instead of
>>>   putting them on top of our minimal GET/PUT/DELETE API definition.
>>>   In fact, we could probably separate it into two Internet-Drafts:
>>>   one for the 'RESTful folders' API which is our simplification of
>>>   WebDAV, and a second one for OAuth/ETags/CORS/WebFinger on top of
>>>   either WebDAV or 'RESTful folders' or whatever other HTTP-based
>>>   API you want.
>>
>>
>> There is one requirement that all synchronisers need to handle:
>> moving resources. In this spec the alternative of a WebDAV MOVE is
>> not specified.
>>
>> It is true that a MOVE could be replaced with a DELETE + UPLOAD but
>> that is not acceptable in most cases due to the inefficiency of such
>> operation (cpu, bandwidth consumption ...)
>>
>> Is there a plan to support such basic feature?
>>
>> From the current remoteStorage spec, the ownCloud sync protocol can
>> be implemented. The missing bit is tracking those remote moves.
> I agree with Hugo that MOVE is useful, sometimes if you just rename a
> file it will be perfect. But the question I have is that whether we
> need to make two documents? Multiple choices is not good for
> standardization. In my view, webdav is something that we need to have
> a very clear decision on whether to consider it or not.
>
> Regards, Linhui
>>
>>
>> Cheers
>>
>>> On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador
>>> <[e-mail]> wrote:
>>>> __
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
>>>>> Hi
>>>>>
>>>>> On 周三, 12月 2, 2015 at 17:56,  Hugo González Labrador
>>>>> <[e-mail]> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> 2015-12-02 5:14 GMT+08:00 Hugo González Labrador
>>>>>>> <[e-mail]>:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> from my point of view the remoteStorage project addresses a
>>>>>>>> subset of the use cases of the  WebDAV specification.
>>>>>>>>
>>>>>>>> The main difference I can observe is that the specification is
>>>>>>>> built on REST instead of XML-based communication.
>>>>>>>>
>>>>>>>> I personally like much more REST than WebDAV because it is more
>>>>>>>> developer friendly and it is faster to develop. Maybe the
>>>>>>>> remoteStorage API becomes the next WebDAV :)
>>>>>>>>
>>>>>>>> However, the remoteStorage API does not provide a way of
>>>>>>>> synchronising data, this task is delegated to the developers.
>>>>>>>> Is there a plan to provide such feature based on the outcome of
>>>>>>>> this draft?
>>>>>>> I'm a little bit confused why you say the remoteStorage does not
>>>>>>> provide that. Is that because remoteStorage does not perform
>>>>>>> like a typical sync services (e.g. dropbox...) or you are saying
>>>>>>> something else?
>>>>>> Yes, because it does not offer synchronisation capabilities.
>>>>> Got it. And What I am wondering is that do we need to include
>>>>> those capabilities in a base specifications. Since it is hard to
>>>>> standardize the capabilities like dedup or delta. Maybe a better
>>>>> way is to define those in a separate specification,
>>>>
>>>
>>> Thanks for giving these examples - so by
'synchronisation capabilities' you mean 1) deduplication and 2) delta
updates? Anything else or is that an exhaustive list?
>>>>
>>>>> something like extensions. For a base document, we just need to
>>>>> define how to perform sync operations.
>>>>
>>>>
>>>> Yes, I agree that may be  an extension of this draft could handle
>>>> the synchronisation part.
>>>>
>>>>
>>>>
>>>
>>> Our Internet-Draft is heavily focused on the world wide web, whose
>>> URLs are not content-addressable, we can't change that. So that
>>> architecture is not very friendly to deduplication, compared to for
>>> instance BitTorrent. As you already said, developers can still
>>> dedupe on the server-side when storing blobs to disk, and can also
>>> dedupe on the client side before creating the resources the client
>>> uploads. As far as I know, delta updates are not supported by the
>>> ETag system - you cannot do a range request to find out if certain
>>> bytes within a document have changed. However, the folder system we
>>> define does encourage you, for instance when you develop a To Do
>>> List app, put each task on its own document, and then query the
>>> folder to see which of them changed, instead of putting them all in
>>> one big document and retrieving the whole document each time.
>>>
>>> Cheers, Michiel.
>>>
>>>>
>>>>>>
>>>>>>>>
>>>>>>>> BTW, I want to introduce ClawIO (http://clawio.github.io), a
>>>>>>>> research project to benchmark different synchronisation
>>>>>>>> protocols against different data backends with special
>>>>>>>> attention to provide a common sync API.
>>>>>>>>
>>>>>>>> A common API for different sync protocols is being created
>>>>>>>> based on the architecture specified in this draft (control and
>>>>>>>> data servers) and the
>>>>>>> I cannot find a distributed architecture in this draft. It seems
>>>>>>> that they handle metadata and content data together, just like a
>>>>>>> normal web service.
>>>>>>
>>>>>> ClawIO is fully distributed. Every logical unit is a different
>>>>>> server than be scaled out. Data and Metadata channels are
>>>>>> independent from each other and reside on different servers.
>>>>> That is widely employed in popular sync services. And that is also
>>>>> beneficial to privacy to some extent. But in the context of sync
>>>>> in browser (which is mainly considered in the remoteStorage), I
>>>>> don't know whether this is reasonable. But I think we should
>>>>> deploy distributed architecture although it will make things
>>>>> complicated.
>>>>
>>>>
>>>> Of course, the remoteStorage is targeted to browsers, so syncing
>>>> does not make too much sense in this case. With the rise of Linux
>>>> container micro-service based architectures, the deployment of
>>>> such highly complex systems should become easier and faster.
>>>>
>>>> Best,
>>>>
>>>>
>>>> Hugo
>>>>
>>>>
>>>>> Regards, Linhui
>>>>>>
>>>>>>
>>>>>>> Regards, Linhui
>>>>>>>> one from the CERN EOS project (management, disk and queue
>>>>>>>> servers).
>>>>>>>>
>>>>>>>> The Phase I has implemented the ownCloud Sync Protocol and
>>>>>>>> Phase II will implement the SeaFile Sync Protocol. The choice
>>>>>>>> of these protocols among others is because they are really
>>>>>>>> opposed to each other in terms of syncing (delta vs non-delta,
>>>>>>>> state-based vs log/event/git-based sync …), so finding a common
>>>>>>>> approach is more challenging.
>>>>>>>>
>>>>>>>> Providing a base specification/architecture to measure the
>>>>>>>> feasibility of this draft is one of the objectives of the
>>>>>>>> project.
>>>>>>>>
>>>>>>>> I believe that the work being done here and in ClawIO are
>>>>>>>> supplementary to each other and I think mutual collaboration
>>>>>>>> could be beneficial for both sides.
>>>>>>>>
>>>>>>>> Also, if there is interest, the remoteStorage API can be added
>>>>>>>> to ClawIO.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Hugo Gonzalez Labrador
>>>>>>>>
>>>>>>>> On Tue, Dec 1, 2015, at 09:00 PM, [e-mail]
>>>>>>>> wrote:
>>>>>>>> > Send Storagesync mailing list submissions to
>>>>>>>> > [e-mail]
>>>>>>>> >
>>>>>>>> > To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync or, via
>>>>>>>> > email, send a message with subject or body 'help' to
>>>>>>>> > [e-mail]
>>>>>>>> >
>>>>>>>> > You can reach the person managing the list at       storagesync-
>>>>>>>> > [e-mail]
>>>>>>>> >
>>>>>>>> > When replying, please edit your Subject line so it is more
>>>>>>>> > specific than "Re: Contents of Storagesync digest..." Today's
>>>>>>>> > Topics:
>>>>>>>> >
>>>>>>>> >1. New version of draft-dejong-remotestorage    Internet-Draft
>>>>>>>> >available (Michiel de Jong)    2. Re: New version of draft-dejong-
>>>>>>>> >remotestorage Internet-Draft       available (Gihan Dias)
>>>>>>>> >3. Re: New version of draft-dejong-remotestorage Internet-
>>>>>>>> >Draft       available (Fei Song)
>>>>>>>> > _______________________________________________
>>>>>>>> > Storagesync mailing list [e-mail]
>>>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync Email had 3
>>>>>>>> > attachments:
>>>>>>>> > + [Storagesync] New version of draft-dejong-remotestorage Internet-
>>>>>>>> >   Draft available   2k (message/rfc822)
>>>>>>>> > + Re: [Storagesync] New version of draft-dejong-remotestorage
>>>>>>>> >   Internet-Draft available   1k (message/rfc822)
>>>>>>>> > + Re: [Storagesync] New version of draft-dejong-remotestorage
>>>>>>>> >   Internet-Draft available   2k (message/rfc822)
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Storagesync mailing list [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/storagesync
>>>>>>
>>>>
>>