Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Linhui Sun <[e-mail]>]

Hi

I've missed some part of your message yesterday, please see my comments
inline.

2015-12-02 20:30 GMT+08:00 Michiel de Jong <[e-mail]>:

> Hi all!
>
> Thanks for all your reactions to the remoteStorage Internet-Draft.
>
> We get the question about WebDAV a lot, in the next version we should add
> a remark about it in the intro. The folder descriptions returned when you
> GET a URL that ends with a / are indeed a deviation from the XML returned
> by WebDAV server, and this is just because nowadays JSON is easier to use
> than XML for developers, both client-side and server-side.
>
> The fact that we don't require servers to support WebDAV's custom verbs
> like PROPPATCH etc. is for three reasons:
> 1) it's a lot of work to implement this without using an existing WebDAV
> library
> 2) in practice, a lot of WebDAV servers get it wrong, or don't implement
> all of WebDAV. It's very annoying for client implementers to have to deal
> with servers that e.g. chose not to implement LOCK and UNLOCK.
> 3) we don't really need all these advanced features on top of standard
> HTTP, just supporting GET/PUT/DELETE for resources, and adding a simple
> folder description format, is enough for most applications.
>
> Other than that, the remoteStorage Internet-Draft specifies a *lot* more
> than just how each HTTP verb should behave:
> * requiring support for OAuth implicit-grant flow
> * requiring ETag support and nested versioning (i.e. the folder's ETag
> changes if anything within that folder changes)
> * requiring CORS headers
> * requiring a WebFinger announcement for service discovery
>
> It would be easy to add these three things on top of WebDAV, instead of
> putting them on top of our minimal GET/PUT/DELETE API definition. In fact,
> we could probably separate it into two Internet-Drafts: one for the
> 'RESTful folders' API which is our simplification of WebDAV, and a second
> one for OAuth/ETags/CORS/WebFinger on top of either WebDAV or 'RESTful
> folders' or whatever other HTTP-based API you want.
>
> On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador <
> [e-mail]> wrote:
>
>>
>>
>>
>> On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
>>
>> Hi
>>
>> On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador <
>> [e-mail]> wrote:
>>
>>
>>
>>
>> On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
>>
>> Hi
>>
>> 2015-12-02 5:14 GMT+08:00 Hugo González Labrador <[e-mail]>:
>>
>> Hi,
>>
>> from my point of view the remoteStorage project addresses a subset of
>> the use cases of the  WebDAV specification.
>>
>> The main difference I can observe is that the specification is built on
>> REST instead of XML-based communication.
>>
>>
>> I personally like much more REST than WebDAV because it is more
>> developer friendly and it is faster to develop.
>>  Maybe the remoteStorage API becomes the next WebDAV :)
>>
>> However, the remoteStorage API does not provide a way of synchronising
>> data, this task is delegated to the developers.
>> Is there a plan to provide such feature based on the outcome of this
>> draft?
>>
>> I'm a little bit confused why you say the remoteStorage does not provide
>> that. Is that because remoteStorage does not perform like a typical sync
>> services (e.g. dropbox...) or you are saying something else?
>>
>> Yes, because it does not offer synchronisation capabilities.
>>
>> Got it. And What I am wondering is that do we need to include those
>> capabilities in a base specifications. Since it is hard to standardize the
>> capabilities like dedup or delta. Maybe a better way is to define those in
>> a separate specification,
>>
>>
> Thanks for giving these examples - so by 'synchronisation capabilities'
> you mean 1) deduplication and 2) delta updates? Anything else or is that an
> exhaustive list?
>
Yes and something like chunking (the one I've mentioned in the previous
message) and bundling. They are typically client capabilities (or referred
as service capabilities). These should be beyond the basic document like
the remoteStorage, but is important and useful to be defined in a companion
document.

> something like extensions. For a base document, we just need to define how
>> to perform sync operations.
>>
>>
>> Yes, I agree that may be an extension of this draft could handle the
>> synchronisation part.
>>
>>
>
> Our Internet-Draft is heavily focused on the world wide web, whose URLs
> are not content-addressable, we can't change that. So that architecture is
> not very friendly to deduplication, compared to for instance BitTorrent. As
> you already said, developers can still dedupe on the server-side when
> storing blobs to disk, and can also dedupe on the client side before
> creating the resources the client uploads.
>
IMO, performing dedup. inside server or client itself is not interesting.
What we want for a sync service is to reduce the number of transmitting
packets between client and server. That requires both client and server
should be aware of which packet should be transmitted. And I think
capabilities like dedup. has nothing to do with whether you use a
distributed architecture.

>
> As far as I know, delta updates are not supported by the ETag system - you
> cannot do a range request to find out if certain bytes within a document
> have changed. However, the folder system we define does encourage you, for
> instance when you develop a To Do List app, put each task on its own
> document, and then query the folder to see which of them changed, instead
> of putting them all in one big document and retrieving the whole document
> each time.
>
That's a delta update in essence, but is done by user behavior. And could
we make the "query the folder" something like server push (similar issue in
webpush WG)? When there is a change, the server will notify the client.

Regards,
Linhui

>
>
> Cheers,
> Michiel.
>
>
>>
>>
>> BTW, I want to introduce ClawIO ( <http://clawio.github.io>
>> http://clawio.github.io), a research
>> project to benchmark different synchronisation protocols against
>> different data backends with special attention to provide a common sync
>> API.
>>
>> A common API for different sync protocols is being created based on the
>> architecture specified in this draft (control and data servers) and the
>>
>> I cannot find a distributed architecture in this draft. It seems that
>> they handle metadata and content data together, just like a normal web
>> service.
>>
>>
>> ClawIO is fully distributed. Every logical unit is a different server
>> than be scaled out. Data and Metadata channels are independent from each
>> other and reside on different servers.
>>
>> That is widely employed in popular sync services. And that is also
>> beneficial to privacy to some extent. But in the context of sync in browser
>> (which is mainly considered in the remoteStorage), I don't know whether
>> this is reasonable. But I think we should deploy distributed architecture
>> although it will make things complicated.
>>
>>
>> Of course, the remoteStorage is targeted to browsers, so syncing does not
>> make too much sense in this case.
>> With the rise of Linux container micro-service based architectures, the
>> deployment of  such highly complex systems should become easier and faster.
>>
>> Best,
>>
>> Hugo
>>
>>
>> Regards,
>> Linhui
>>
>>
>>
>>
>> Regards,
>> Linhui
>>
>> one from the CERN EOS project (management, disk and queue servers).
>>
>> The Phase I has implemented the ownCloud Sync Protocol and Phase II will
>> implement the SeaFile Sync Protocol.
>> The choice of these protocols among others is because they are really
>> opposed to each other in terms of syncing (delta vs non-delta,
>> state-based vs log/event/git-based sync …), so finding a common approach
>> is more challenging.
>>
>> Providing a base specification/architecture to measure the feasibility
>> of this draft is one of the objectives of the project.
>>
>> I believe that the work being done here and in ClawIO are supplementary
>> to each other and I think mutual collaboration could be beneficial for
>> both sides.
>>
>> Also, if there is interest, the remoteStorage API can be added to
>> ClawIO.
>>
>> Best regards,
>>
>> Hugo Gonzalez Labrador
>>
>> On Tue, Dec 1, 2015, at 09:00 PM, [e-mail] wrote:
>> > Send Storagesync mailing list submissions to
>> >       [e-mail]
>> >
>> > To subscribe or unsubscribe via the World Wide Web, visit
>> >        <https://www.ietf.org/mailman/listinfo/storagesync>
>> https://www.ietf.org/mailman/listinfo/storagesync
>> > or, via email, send a message with subject or body 'help' to
>> >       [e-mail]
>> >
>> > You can reach the person managing the list at
>> >       [e-mail]
>> >
>> > When replying, please edit your Subject line so it is more specific
>> > than "Re: Contents of Storagesync digest..."
>> > Today's Topics:
>> >
>> >    1. New version of draft-dejong-remotestorage    Internet-Draft
>> >       available (Michiel de Jong)
>> >    2. Re: New version of draft-dejong-remotestorage Internet-Draft
>> >       available (Gihan Dias)
>> >    3. Re: New version of draft-dejong-remotestorage Internet-Draft
>> >       available (Fei Song)
>> > _______________________________________________
>> > Storagesync mailing list
>> > [e-mail]
>> > <https://www.ietf.org/mailman/listinfo/storagesync>
>> https://www.ietf.org/mailman/listinfo/storagesync
>> > Email had 3 attachments:
>> > + [Storagesync] New version of draft-dejong-remotestorage
>> > Internet-Draft available
>> >   2k (message/rfc822)
>> > + Re: [Storagesync] New version of draft-dejong-remotestorage
>> > Internet-Draft available
>> >   1k (message/rfc822)
>> > + Re: [Storagesync] New version of draft-dejong-remotestorage
>> > Internet-Draft available
>> >   2k (message/rfc822)
>>
>> _______________________________________________
>> Storagesync mailing list
>> [e-mail]
>> <https://www.ietf.org/mailman/listinfo/storagesync>
>> https://www.ietf.org/mailman/listinfo/storagesync
>>
>>
>>
>>
>>
>
>