Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

[Linhui Sun <[e-mail]>]

2015-12-03 17:49 GMT+08:00 Hugo González Labrador <[e-mail]>:

> What is the best way to collect the advantages and drawbacks of WebDAV vs
> remoteStorage ?
>
> I propose to create a GitHub instead of/in addition to specify the
> differences here. I think is clear to show this info in something like a
> table on GitHub instead of in a series of mail messages.
>
That is a good way to discuss, but I think we should not saying the pros or
cons, it's just about applicability here. A discussion about whether it is
suitable to build on WebDAV is better.

Regards,
Linhui

>
> I can create the repo if it is okay for you.
>
> Best,Hugo
>
>
> On Thu, Dec 3, 2015, at 10:31 AM, [e-mail] wrote:
>
> Great！
>
>
>
> -----原始邮件-----
> *发件人:* "Michiel de Jong" <[e-mail]>
> *发送时间:* 2015年12月3日 星期四
> *收件人:* [e-mail]
> *抄送:* "Linhui Sun" <[e-mail]>, storagesync <
> [e-mail]>, fkooman <[e-mail]>, "Hugo González
> Labrador" <[e-mail]>
> *主题:* Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1
>
> Yes! I'll discuss with Francois how we can make these boundaries clearer.
> Maybe we can split along these lines:
> * upload/manipulate/query/download protocol (e.g. WebDAV or the
> GET/PUT/DELETE operations we currently define)
> * CORS or other cross-origin access mechanism, OAUTH or other auth
> mechanism, WebFinger or other discovery mechanism, ETag or other versioning
> mechanism
> * chunking/deduping/delta-updates
>
> And the remoteStorage spec should probably only define the middle part.
>
>
> On Thu, Dec 3, 2015 at 9:44 AM, <[e-mail]> wrote:
>
> Hi Michiel and Linhui,
>
> I think it will be good to have a boundary for this draft and leave some
> work for the applicaiton layer~
>
> -----原始邮件-----
> *发件人:* "Michiel de Jong" <[e-mail]>
> *发送时间:* 2015年12月2日 星期三
> *收件人:* "Linhui Sun" <[e-mail]>
> *抄送:* storagesync <[e-mail]>, fkooman <[e-mail]>,
> "Hugo González Labrador" <[e-mail]>
> *主题:* Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1
>
> Ah sure, that's entirely appropriate, remoteStorage treats both the
> Content-Type header value and the actual body of a document as opaque
> strings of bytes. It doesn't "care" if you use it to store only data blocks
> that are chunks of something else.
> For instance, you could have a folder on a user's storage that contains
> only inode-like JSON-documents, which list the URLs of binary documents
> that make up 1Kb blocks of the actual data. Nice for deduping, delta
> updates, and also renaming files without reuploading their content.
>
> But yeah, the argument is that *how* to create and manage these chunks, is
> then still left up to the application developer (or to another spec on top
> of the remoteStorage spec).
>
>
> Cheers,
> Michiel.
>
> On Wed, Dec 2, 2015 at 3:29 PM, Linhui Sun <[e-mail]> wrote:
>
> Hi
>
> 2015-12-02 22:05 GMT+08:00 Michiel de Jong <[e-mail]>:
>
> Cool! I created https://github.com/remotestorage/spec/issues/137 about
> the need for  MOVE verb.
> Application-level chunking is partially supported by HTTP itself through
> `Content-Range` headers (although it's not clear whether these are allowed
> on PUT requests as well as on GET requests, see
> https://github.com/remotestorage/spec/issues/131). The problem is that
> HTTP defines versioning at the document level, you cannot ask a server to
> produce or check an ETag for a specific byte-range of a document, only for
> the entire document.
>
>
> Actually what I'm saying is a chunking before transmitting (using http),
> in this way, they are treated as individual documents from the standpoint
> of http. But I don't know whether this is appropriate for remoteStorage,
> just a comment.
>
> Regards,
> Linhui
>
>
> A comparison document sounds good! See also
> http://www.servicedenuages.fr/en/generic-storage-ecosystem.
>
>
> Cheers,
> Michiel.
>
>
> On Wed, Dec 2, 2015 at 2:32 PM, Linhui Sun <[e-mail]> wrote:
>
> That's cool for me, a separate section for this might make sense.
>
> Another thing is do we need to include an application-layer chunking here
> (not just for a browser sync), since if we want to further extend other
> capabilities it is essential.
>
> Regards,
> Linhui
>
> 2015-12-02 21:03 GMT+08:00 Hugo González Labrador <[e-mail]>:
>
>
> I propose to come up with a list of advantages and disadvantages of using
> WebDAV and compare them against a JSON/REST based approach, like
> remoteStorage.
>
> Does it sound good ?
>
>
> On Wed, Dec 2, 2015, at 01:59 PM, Linhui Sun wrote:
>
>
>
> 2015-12-02 20:43 GMT+08:00 Hugo González Labrador <[e-mail]>:
>
>
>
>
>
>
> On Wed, Dec 2, 2015, at 01:30 PM, Michiel de Jong wrote:
>
> Hi all!
> Thanks for all your reactions to the remoteStorage Internet-Draft.
>
> We get the question about WebDAV a lot, in the next version we should add
> a remark about it in the intro. The folder descriptions returned when you
> GET a URL that ends with a / are indeed a deviation from the XML returned
> by WebDAV server, and this is just because nowadays JSON is easier to use
> than XML for developers, both client-side and server-side.
>
>
>
> I totally agree here, this was going to happen soon or later and it is
> really appreciated.
>
>
>
> The fact that we don't require servers to support WebDAV's custom verbs
> like PROPPATCH etc. is for three reasons:
> 1) it's a lot of work to implement this without using an existing WebDAV
> library
> 2) in practice, a lot of WebDAV servers get it wrong, or don't implement
> all of WebDAV. It's very annoying for client implementers to have to deal
> with servers that e.g. chose not to implement LOCK and UNLOCK.
> 3) we don't really need all these advanced features on top of standard
> HTTP, just supporting GET/PUT/DELETE for resources, and adding a simple
> folder description format, is enough for most applications.
>
> Other than that, the remoteStorage Internet-Draft specifies a *lot* more
> than just how each HTTP verb should behave:
> * requiring support for OAuth implicit-grant flow
> * requiring ETag support and nested versioning (i.e. the folder's ETag
> changes if anything within that folder changes)
> * requiring CORS headers
> * requiring a WebFinger announcement for service discovery
> It would be easy to add these three things on top of WebDAV, instead of
> putting them on top of our minimal GET/PUT/DELETE API definition. In fact,
> we could probably separate it into two Internet-Drafts: one for the
> 'RESTful folders' API which is our simplification of WebDAV, and a second
> one for OAuth/ETags/CORS/WebFinger on top of either WebDAV or 'RESTful
> folders' or whatever other HTTP-based API you want.
>
>
>
> There is one requirement that all synchronisers need to handle: moving
> resources. In this spec the alternative of a WebDAV MOVE is not specified.
>
> It is true that a MOVE could be replaced with a DELETE + UPLOAD but that
> is not acceptable in most cases due to the inefficiency of such operation
> (cpu, bandwidth consumption ...)
>
> Is there a plan to support such basic feature?
>
> From the current remoteStorage spec, the ownCloud sync protocol can be
> implemented. The missing bit is tracking those remote moves.
>
> I agree with Hugo that MOVE is useful, sometimes if you just rename a file
> it will be perfect. But the question I have is that whether we need to make
> two documents? Multiple choices is not good for standardization. In my
> view, webdav is something that we need to have a very clear decision on
> whether to consider it or not.
>
> Regards,
> Linhui
>
>
>
> Cheers
>
>
> On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador <
> [e-mail]> wrote:
>
>
>
>
>
>
> On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
>
> Hi
>
> On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador <[e-mail]>
> wrote:
>
>
>
>
> On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
>
> Hi
>
> 2015-12-02 5:14 GMT+08:00 Hugo González Labrador <[e-mail]>:
>
> Hi,
>
> from my point of view the remoteStorage project addresses a subset of
> the use cases of the  WebDAV specification.
>
> The main difference I can observe is that the specification is built on
> REST instead of XML-based communication.
>
>
> I personally like much more REST than WebDAV because it is more
> developer friendly and it is faster to develop.
>  Maybe the remoteStorage API becomes the next WebDAV :)
>
> However, the remoteStorage API does not provide a way of synchronising
> data, this task is delegated to the developers.
> Is there a plan to provide such feature based on the outcome of this
> draft?
>
> I'm a little bit confused why you say the remoteStorage does not provide
> that. Is that because remoteStorage does not perform like a typical sync
> services (e.g. dropbox...) or you are saying something else?
>
> Yes, because it does not offer synchronisation capabilities.
>
> Got it. And What I am wondering is that do we need to include those
> capabilities in a base specifications. Since it is hard to standardize the
> capabilities like dedup or delta. Maybe a better way is to define those in
> a separate specification,
>
>
>
>
> Thanks for giving these examples - so by 'synchronisation capabilities'
> you mean 1) deduplication and 2) delta updates? Anything else or is that an
> exhaustive list?
>
>
>
> something like extensions. For a base document, we just need to define how
> to perform sync operations.
>
>
>
> Yes, I agree that may be an extension of this draft could handle the
> synchronisation part.
>
>
>
>
>
> Our Internet-Draft is heavily focused on the world wide web, whose URLs
> are not content-addressable, we can't change that. So that architecture is
> not very friendly to deduplication, compared to for instance BitTorrent. As
> you already said, developers can still dedupe on the server-side when
> storing blobs to disk, and can also dedupe on the client side before
> creating the resources the client uploads.
> As far as I know, delta updates are not supported by the ETag system - you
> cannot do a range request to find out if certain bytes within a document
> have changed. However, the folder system we define does encourage you, for
> instance when you develop a To Do List app, put each task on its own
> document, and then query the folder to see which of them changed, instead
> of putting them all in one big document and retrieving the whole document
> each time.
>
> Cheers,
> Michiel.
>
>
>
>
>
>
>
> BTW, I want to introduce ClawIO ( <http://clawio.github.io/>
> http://clawio.github.io), a research
> project to benchmark different synchronisation protocols against
> different data backends with special attention to provide a common sync
> API.
>
> A common API for different sync protocols is being created based on the
> architecture specified in this draft (control and data servers) and the
>
> I cannot find a distributed architecture in this draft. It seems that they
> handle metadata and content data together, just like a normal web service.
>
>
> ClawIO is fully distributed. Every logical unit is a different server than
> be scaled out. Data and Metadata channels are independent from each other
> and reside on different servers.
>
> That is widely employed in popular sync services. And that is also
> beneficial to privacy to some extent. But in the context of sync in browser
> (which is mainly considered in the remoteStorage), I don't know whether
> this is reasonable. But I think we should deploy distributed architecture
> although it will make things complicated.
>
>
>
> Of course, the remoteStorage is targeted to browsers, so syncing does not
> make too much sense in this case.
> With the rise of Linux container micro-service based architectures, the
> deployment of  such highly complex systems should become easier and faster.
>
> Best,
>
>
> Hugo
>
>
>
> Regards,
> Linhui
>
>
>
>
> Regards,
> Linhui
>
> one from the CERN EOS project (management, disk and queue servers).
>
> The Phase I has implemented the ownCloud Sync Protocol and Phase II will
> implement the SeaFile Sync Protocol.
> The choice of these protocols among others is because they are really
> opposed to each other in terms of syncing (delta vs non-delta,
> state-based vs log/event/git-based sync …), so finding a common approach
> is more challenging.
>
> Providing a base specification/architecture to measure the feasibility
> of this draft is one of the objectives of the project.
>
> I believe that the work being done here and in ClawIO are supplementary
> to each other and I think mutual collaboration could be beneficial for
> both sides.
>
> Also, if there is interest, the remoteStorage API can be added to
> ClawIO.
>
> Best regards,
>
> Hugo Gonzalez Labrador
>
> On Tue, Dec 1, 2015, at 09:00 PM, [e-mail] wrote:
> > Send Storagesync mailing list submissions to
> >       [e-mail]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >        <https://www.ietf.org/mailman/listinfo/storagesync>
> https://www.ietf.org/mailman/listinfo/storagesync
> > or, via email, send a message with subject or body 'help' to
> >       [e-mail]
> >
> > You can reach the person managing the list at
> >       [e-mail]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Storagesync digest..."
> > Today's Topics:
> >
> >    1. New version of draft-dejong-remotestorage    Internet-Draft
> >       available (Michiel de Jong)
> >    2. Re: New version of draft-dejong-remotestorage Internet-Draft
> >       available (Gihan Dias)
> >    3. Re: New version of draft-dejong-remotestorage Internet-Draft
> >       available (Fei Song)
> > _______________________________________________
> > Storagesync mailing list
> > [e-mail]
> > <https://www.ietf.org/mailman/listinfo/storagesync>
> https://www.ietf.org/mailman/listinfo/storagesync
> > Email had 3 attachments:
> > + [Storagesync] New version of draft-dejong-remotestorage
> > Internet-Draft available
> >   2k (message/rfc822)
> > + Re: [Storagesync] New version of draft-dejong-remotestorage
> > Internet-Draft available
> >   1k (message/rfc822)
> > + Re: [Storagesync] New version of draft-dejong-remotestorage
> > Internet-Draft available
> >   2k (message/rfc822)
>
> _______________________________________________
> Storagesync mailing list
> [e-mail]
> <https://www.ietf.org/mailman/listinfo/storagesync>
> https://www.ietf.org/mailman/listinfo/storagesync
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Storagesync mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/storagesync
>
>
>