Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Hugo González Labrador <[e-mail]>]

What is the best way to collect the advantages and drawbacks of WebDAV
vs remoteStorage ?

I propose to create a GitHub instead of/in addition to specify the
differences here. I think is clear to show this info in something like a
table on GitHub instead of in a series of mail messages.

I can create the repo if it is okay for you.

Best,Hugo


On Thu, Dec 3, 2015, at 10:31 AM, [e-mail] wrote:
> Great！
>
>
>> -----原始邮件----- *发件人:* "Michiel de Jong" <[e-mail]>
>> *发送时间:* 2015年12月3日 星期四 *收件人:* [e-mail] *抄送:* "Linhui Sun"
>> <[e-mail]>, storagesync <[e-mail]>, fkooman
>> <[e-mail]>, "Hugo González Labrador" <[e-mail]>
>> *主题:* Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1
>>
>> Yes! I'll discuss with Francois how we can make these boundaries
>> clearer. Maybe we can split along these lines:
>> * upload/manipulate/query/download protocol (e.g. WebDAV or the
>>   GET/PUT/DELETE operations we currently define)
>> * CORS or other cross-origin access mechanism, OAUTH or other auth
>>   mechanism, WebFinger or other discovery mechanism, ETag or other
>>   versioning mechanism
>> * chunking/deduping/delta-updates
>>
>> And the remoteStorage spec should probably only define the
>> middle part.
>>
>>
>> On Thu, Dec 3, 2015 at 9:44 AM, <[e-mail]> wrote:
>>> Hi Michiel and Linhui,


>>> I think it will be good to have a boundary for this draft and leave
>>> some work for the applicaiton layer~


>>>> -----原始邮件----- *发件人:* "Michiel de Jong" <[e-mail]>
>>>> *发送时间:* 2015年12月2日 星期三 *收件人:* "Linhui Sun" <[e-mail]>
>>>> *抄送:* storagesync <[e-mail]>, fkooman
>>>> <[e-mail]>, "Hugo González Labrador"
>>>> <[e-mail]> *主题:* Re: [Storagesync] Storagesync Digest,
>>>> Vol 5, Issue 1
>>>>
>>>> Ah sure, that's entirely appropriate, remoteStorage treats both the
>>>> Content-Type header value and the actual body of a document as
>>>> opaque strings of bytes. It doesn't "care" if you use it to store
>>>> only data blocks that are chunks of something else. For instance,
>>>> you could have a folder on a user's storage that contains only inode-
>>>> like JSON-documents, which list the URLs of binary documents that
>>>> make up 1Kb blocks of the actual data. Nice for deduping, delta
>>>> updates, and also renaming files without reuploading their content.
>>>>
>>>> But yeah, the argument is that *how* to create and manage these
>>>> chunks, is then still left up to the application developer (or to
>>>> another spec on top of the remoteStorage spec).
>>>>
>>>>
>>>> Cheers, Michiel.
>>>>
>>>> On Wed, Dec 2, 2015 at 3:29 PM, Linhui Sun <[e-mail]>
>>>> wrote:
>>>>> Hi
>>>>>
>>>>> 2015-12-02 22:05 GMT+08:00 Michiel de Jong <[e-mail]>:
>>>>>> Cool! I created https://github.com/remotestorage/spec/issues/137
>>>>>> about the need for  MOVE verb. Application-level chunking is
>>>>>> partially supported by HTTP itself through `Content-Range`
>>>>>> headers (although it's not clear whether these are allowed on PUT
>>>>>> requests as well as on GET requests, see
>>>>>> https://github.com/remotestorage/spec/issues/131). The problem is
>>>>>> that HTTP defines versioning at the document level, you cannot
>>>>>> ask a server to produce or check an ETag for a specific byte-
>>>>>> range of a document, only for the entire document.
>>>>>
>>>>> Actually what I'm saying is a chunking before transmitting (using
>>>>> http), in this way, they are treated as individual documents from
>>>>> the standpoint of http. But I don't know whether this is
>>>>> appropriate for remoteStorage, just a comment.
>>>>>
>>>>> Regards, Linhui
>>>>>>
>>>>>> A comparison document sounds good! See also
>>>>>> http://www.servicedenuages.fr/en/generic-storage-ecosystem.
>>>>>>
>>>>>>
>>>>>> Cheers, Michiel.
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 2:32 PM, Linhui Sun <[e-mail]>
>>>>>> wrote:
>>>>>>> That's cool for me, a separate section for this might make
>>>>>>> sense.
>>>>>>>
>>>>>>> Another thing is do we need to include an application-layer
>>>>>>> chunking here (not just for a browser sync), since if we want to
>>>>>>> further extend other capabilities it is essential.
>>>>>>>
>>>>>>> Regards, Linhui
>>>>>>>
>>>>>>> 2015-12-02 21:03 GMT+08:00 Hugo González Labrador
>>>>>>> <[e-mail]>:
>>>>>>>> __
>>>>>>>> I propose to come up with a list of advantages and
>>>>>>>> disadvantages of using WebDAV and compare them against a
>>>>>>>> JSON/REST based approach, like remoteStorage.
>>>>>>>>
>>>>>>>> Does it sound good ?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015, at 01:59 PM, Linhui Sun wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-12-02 20:43 GMT+08:00 Hugo González Labrador
>>>>>>>>> <[e-mail]>:
>>>>>>>>>> __
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015, at 01:30 PM, Michiel de Jong wrote:
>>>>>>>>>>> Hi all! Thanks for all your reactions to the remoteStorage
>>>>>>>>>>> Internet-Draft.
>>>>>>>>>>>
>>>>>>>>>>> We get the question about WebDAV a lot, in the next version
>>>>>>>>>>> we should add a remark about it in the intro. The folder
>>>>>>>>>>> descriptions returned when you GET a URL that ends with a /
>>>>>>>>>>> are indeed a deviation from the XML returned by WebDAV
>>>>>>>>>>> server, and this is just because nowadays JSON is easier to
>>>>>>>>>>> use than XML for developers, both client-side and server-
>>>>>>>>>>> side.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I totally agree here, this was going to happen soon or later
>>>>>>>>>> and it is really appreciated.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> The fact that we don't require servers to support WebDAV's
>>>>>>>>>>> custom verbs like PROPPATCH etc. is for three reasons:
>>>>>>>>>>> 1) it's a lot of work to implement this without using an
>>>>>>>>>>>    existing WebDAV library
>>>>>>>>>>> 2) in practice, a lot of WebDAV servers get it wrong, or
>>>>>>>>>>>    don't implement all of WebDAV. It's very annoying for
>>>>>>>>>>>    client implementers to have to deal with servers that
>>>>>>>>>>>    e.g. chose not to implement LOCK and UNLOCK.
>>>>>>>>>>> 3) we don't really need all these advanced features on top
>>>>>>>>>>>    of standard HTTP, just supporting GET/PUT/DELETE for
>>>>>>>>>>>    resources, and adding a simple folder description format,
>>>>>>>>>>>    is enough for most applications.
>>>>>>>>>>>
>>>>>>>>>>> Other than that, the remoteStorage Internet-Draft specifies
>>>>>>>>>>> a *lot* more than just how each HTTP verb should behave:
>>>>>>>>>>> * requiring support for OAuth implicit-grant flow
>>>>>>>>>>> * requiring ETag support and nested versioning (i.e. the
>>>>>>>>>>>   folder's ETag changes if anything within that folder
>>>>>>>>>>>   changes)
>>>>>>>>>>> * requiring CORS headers
>>>>>>>>>>> * requiring a WebFinger announcement for service discovery
>>>>>>>>>>>   It would be easy to add these three things on top of
>>>>>>>>>>>   WebDAV, instead of putting them on top of our minimal
>>>>>>>>>>>   GET/PUT/DELETE API definition. In fact, we could probably
>>>>>>>>>>>   separate it into two Internet-Drafts: one for the 'RESTful
>>>>>>>>>>>   folders' API which is our simplification of WebDAV, and a
>>>>>>>>>>>   second one for OAuth/ETags/CORS/WebFinger on top of either
>>>>>>>>>>>   WebDAV or 'RESTful folders' or whatever other HTTP-based
>>>>>>>>>>>   API you want.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There is one requirement that all synchronisers need to
>>>>>>>>>> handle: moving resources. In this spec the alternative of a
>>>>>>>>>> WebDAV MOVE is not specified.
>>>>>>>>>>
>>>>>>>>>> It is true that a MOVE could be replaced with a DELETE +
>>>>>>>>>> UPLOAD but that is not acceptable in most cases due to the
>>>>>>>>>> inefficiency of such operation (cpu, bandwidth consumption
>>>>>>>>>> ...)
>>>>>>>>>>
>>>>>>>>>> Is there a plan to support such basic feature?
>>>>>>>>>>
>>>>>>>>>> From the current remoteStorage spec, the ownCloud sync
>>>>>>>>>> protocol can be implemented. The missing bit is tracking
>>>>>>>>>> those remote moves.
>>>>>>>>> I agree with Hugo that MOVE is useful, sometimes if you just
>>>>>>>>> rename a file it will be perfect. But the question I have is
>>>>>>>>> that whether we need to make two documents? Multiple choices
>>>>>>>>> is not good for standardization. In my view, webdav is
>>>>>>>>> something that we need to have a very clear decision on
>>>>>>>>> whether to consider it or not.
>>>>>>>>>
>>>>>>>>> Regards, Linhui
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador
>>>>>>>>>>> <[e-mail]> wrote:
>>>>>>>>>>>> __
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador
>>>>>>>>>>>>> <[e-mail]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2015-12-02 5:14 GMT+08:00 Hugo González Labrador
>>>>>>>>>>>>>>> <[e-mail]>:
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> from my point of view the remoteStorage project
>>>>>>>>>>>>>>>> addresses a subset of the use cases of the  WebDAV
>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The main difference I can observe is that the
>>>>>>>>>>>>>>>> specification is built on REST instead of XML-based
>>>>>>>>>>>>>>>> communication.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I personally like much more REST than WebDAV because it
>>>>>>>>>>>>>>>> is more developer friendly and it is faster to develop.
>>>>>>>>>>>>>>>> Maybe the remoteStorage API becomes the next WebDAV :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, the remoteStorage API does not provide a way
>>>>>>>>>>>>>>>> of synchronising data, this task is delegated to the
>>>>>>>>>>>>>>>> developers. Is there a plan to provide such feature
>>>>>>>>>>>>>>>> based on the outcome of this draft?
>>>>>>>>>>>>>>> I'm a little bit confused why you say the remoteStorage
>>>>>>>>>>>>>>> does not provide that. Is that because remoteStorage
>>>>>>>>>>>>>>> does not perform like a typical sync services (e.g.
>>>>>>>>>>>>>>> dropbox...) or you are saying something else?
>>>>>>>>>>>>>> Yes, because it does not offer synchronisation
>>>>>>>>>>>>>> capabilities.
>>>>>>>>>>>>> Got it. And What I am wondering is that do we need to
>>>>>>>>>>>>> include those capabilities in a base specifications. Since
>>>>>>>>>>>>> it is hard to standardize the capabilities like dedup or
>>>>>>>>>>>>> delta. Maybe a better way is to define those in a separate
>>>>>>>>>>>>> specification,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks for giving these examples - so by 'synchronisation
>>>>>>>>>>> capabilities' you mean 1) deduplication and 2) delta
>>>>>>>>>>> updates? Anything else or is that an exhaustive list?
>>>>>>>>>>>>
>>>>>>>>>>>>> something like extensions. For a base document, we just
>>>>>>>>>>>>> need to define how to perform sync operations.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, I agree that may be an extension of this draft could
>>>>>>>>>>>> handle the synchronisation part.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Our Internet-Draft is heavily focused on the world wide web,
>>>>>>>>>>> whose URLs are not content-addressable, we can't change
>>>>>>>>>>> that. So that architecture is not very friendly to
>>>>>>>>>>> deduplication, compared to for instance BitTorrent. As you
>>>>>>>>>>> already said, developers can still dedupe on the server-side
>>>>>>>>>>> when storing blobs to disk, and can also dedupe on the
>>>>>>>>>>> client side before creating the resources the client
>>>>>>>>>>> uploads. As far as I know, delta updates are not supported
>>>>>>>>>>> by the ETag system - you cannot do a range request to find
>>>>>>>>>>> out if certain bytes within a document have changed.
>>>>>>>>>>> However, the folder system we define does encourage you, for
>>>>>>>>>>> instance when you develop a To Do List app, put each task on
>>>>>>>>>>> its own document, and then query the folder to see which of
>>>>>>>>>>> them changed, instead of putting them all in one big
>>>>>>>>>>> document and retrieving the whole document each time.
>>>>>>>>>>>
>>>>>>>>>>> Cheers, Michiel.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> BTW, I want to introduce ClawIO
>>>>>>>>>>>>>>>> (http://clawio.github.io[1]), a research project to
>>>>>>>>>>>>>>>> benchmark different synchronisation protocols against
>>>>>>>>>>>>>>>> different data backends with special attention to
>>>>>>>>>>>>>>>> provide a common sync API.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A common API for different sync protocols is being
>>>>>>>>>>>>>>>> created based on the architecture specified in this
>>>>>>>>>>>>>>>> draft (control and data servers) and the
>>>>>>>>>>>>>>> I cannot find a distributed architecture in this draft.
>>>>>>>>>>>>>>> It seems that they handle metadata and content data
>>>>>>>>>>>>>>> together, just like a normal web service.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ClawIO is fully distributed. Every logical unit is a
>>>>>>>>>>>>>> different server than be scaled out. Data and Metadata
>>>>>>>>>>>>>> channels are independent from each other and reside on
>>>>>>>>>>>>>> different servers.
>>>>>>>>>>>>> That is widely employed in popular sync services. And that
>>>>>>>>>>>>> is also beneficial to privacy to some extent. But in the
>>>>>>>>>>>>> context of sync in browser (which is mainly considered in
>>>>>>>>>>>>> the remoteStorage), I don't know whether this is
>>>>>>>>>>>>> reasonable. But I think we should deploy distributed
>>>>>>>>>>>>> architecture although it will make things complicated.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Of course, the remoteStorage is targeted to browsers, so
>>>>>>>>>>>> syncing does not make too much sense in this case. With the
>>>>>>>>>>>> rise of Linux container micro-service based architectures,
>>>>>>>>>>>> the deployment of  such highly complex systems should
>>>>>>>>>>>> become easier and faster.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hugo
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Regards, Linhui
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards, Linhui
>>>>>>>>>>>>>>>> one from the CERN EOS project (management, disk and
>>>>>>>>>>>>>>>> queue servers).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The Phase I has implemented the ownCloud Sync Protocol
>>>>>>>>>>>>>>>> and Phase II will implement the SeaFile Sync Protocol.
>>>>>>>>>>>>>>>> The choice of these protocols among others is because
>>>>>>>>>>>>>>>> they are really opposed to each other in terms of
>>>>>>>>>>>>>>>> syncing (delta vs non-delta, state-based vs log/event/git-
>>>>>>>>>>>>>>>> based sync …), so finding a common approach is more
>>>>>>>>>>>>>>>> challenging.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Providing a base specification/architecture to measure
>>>>>>>>>>>>>>>> the feasibility of this draft is one of the objectives
>>>>>>>>>>>>>>>> of the project.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I believe that the work being done here and in ClawIO
>>>>>>>>>>>>>>>> are supplementary to each other and I think mutual
>>>>>>>>>>>>>>>> collaboration could be beneficial for both sides.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, if there is interest, the remoteStorage API can
>>>>>>>>>>>>>>>> be added to ClawIO.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hugo Gonzalez Labrador
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Dec 1, 2015, at 09:00 PM, storagesync-
>>>>>>>>>>>>>>>> [e-mail] wrote:
>>>>>>>>>>>>>>>> > Send Storagesync mailing list submissions to
>>>>>>>>>>>>>>>> > [e-mail]
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > To subscribe or unsubscribe via the World Wide Web,
>>>>>>>>>>>>>>>> > visit
>>>>>>>>>>>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync or,
>>>>>>>>>>>>>>>> > via email, send a message with subject or body 'help'
>>>>>>>>>>>>>>>> > to       [e-mail]
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > You can reach the person managing the list at
>>>>>>>>>>>>>>>> > [e-mail]
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > When replying, please edit your Subject line so it is
>>>>>>>>>>>>>>>> > more specific than "Re: Contents of Storagesync
>>>>>>>>>>>>>>>> > digest..." Today's Topics:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >1. New version of draft-dejong-remotestorage    Internet-
>>>>>>>>>>>>>>>> >Draft       available (Michiel de Jong)    2. Re: New
>>>>>>>>>>>>>>>> >version of draft-dejong-remotestorage Internet-Draft
>>>>>>>>>>>>>>>> >available (Gihan Dias)    3. Re: New version of draft-dejong-
>>>>>>>>>>>>>>>> >remotestorage Internet-Draft       available (Fei
>>>>>>>>>>>>>>>> >Song)
>>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>>> > Storagesync mailing list [e-mail]
>>>>>>>>>>>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync
>>>>>>>>>>>>>>>> > Email had 3 attachments:
>>>>>>>>>>>>>>>> > + [Storagesync] New version of draft-dejong-
>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available   2k
>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>> > + Re: [Storagesync] New version of draft-dejong-
>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available   1k
>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>> > + Re: [Storagesync] New version of draft-dejong-
>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available   2k
>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Storagesync mailing list [e-mail]
>>>>>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/storagesync
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Storagesync mailing list [e-mail]
>>> https://www.ietf.org/mailman/listinfo/storagesync



Links:

  1. http://clawio.github.io/