Re: Resumable Uploads

Amos Jeffries <squid3@treenet.co.nz> Sun, 21 April 2013 02:43 UTC

Return-Path: <ietf-http-wg-request@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 42FC221F8B07 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sat, 20 Apr 2013 19:43:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.543
X-Spam-Level:
X-Spam-Status: No, score=-10.543 tagged_above=-999 required=5 tests=[AWL=0.056, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id afyOCrEFfJFm for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sat, 20 Apr 2013 19:43:28 -0700 (PDT)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) by ietfa.amsl.com (Postfix) with ESMTP id 63CAF21F8AE8 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Sat, 20 Apr 2013 19:43:28 -0700 (PDT)
Received: from lists by frink.w3.org with local (Exim 4.72) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1UTkFF-0002Op-TP for ietf-http-wg-dist@listhub.w3.org; Sun, 21 Apr 2013 02:43:13 +0000
Resent-Date: Sun, 21 Apr 2013 02:43:13 +0000
Resent-Message-Id: <E1UTkFF-0002Op-TP@frink.w3.org>
Received: from lisa.w3.org ([128.30.52.41]) by frink.w3.org with esmtp (Exim 4.72) (envelope-from <squid3@treenet.co.nz>) id 1UTkFC-0002OA-Dr for ietf-http-wg@listhub.w3.org; Sun, 21 Apr 2013 02:43:10 +0000
Received: from ip-58-28-153-233.static-xdsl.xnet.co.nz ([58.28.153.233] helo=treenet.co.nz) by lisa.w3.org with esmtp (Exim 4.72) (envelope-from <squid3@treenet.co.nz>) id 1UTkFB-0007TB-GZ for ietf-http-wg@w3.org; Sun, 21 Apr 2013 02:43:10 +0000
Received: from [192.168.2.7] (103-9-43-128.flip.co.nz [103.9.43.128]) by treenet.co.nz (Postfix) with ESMTP id C2A5AE6EC1 for <ietf-http-wg@w3.org>; Sun, 21 Apr 2013 14:42:47 +1200 (NZST)
Message-ID: <51735224.5010405@treenet.co.nz>
Date: Sun, 21 Apr 2013 14:42:44 +1200
From: Amos Jeffries <squid3@treenet.co.nz>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130328 Thunderbird/17.0.5
MIME-Version: 1.0
To: ietf-http-wg@w3.org
References: <CADZbJ9dYFGyrceh03M3B0KdKto7160Dis_geh9um0BhVe1re0g@mail.gmail.com> <EA846138-6537-4709-AC44-149873716E29@mnot.net> <CADZbJ9f4wtaFQEsM_wQn-GaTz+fTKZNyfQk6hXG5OL=Lpkhpcw@mail.gmail.com> <2FD45C0A-1A7C-483B-B8E4-5F300B3EDA66@checkpoint.com> <B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net>
In-Reply-To: <B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Received-SPF: pass client-ip=58.28.153.233; envelope-from=squid3@treenet.co.nz; helo=treenet.co.nz
X-W3C-Hub-Spam-Status: No, score=-2.7
X-W3C-Hub-Spam-Report: AWL=-2.710, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001
X-W3C-Scan-Sig: lisa.w3.org 1UTkFB-0007TB-GZ e56197e3cc1f74ee18c02a24dc1efb68
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Resumable Uploads
Archived-At: <http://www.w3.org/mid/51735224.5010405@treenet.co.nz>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/17445
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On 21/04/2013 2:08 p.m., Mark Nottingham wrote:
> On 21/04/2013, at 5:43 AM, Yoav Nir <ynir@checkpoint.com> wrote:
>
>> On Apr 20, 2013, at 11:30 AM, Felix Geisendörfer <felix@transloadit.com> wrote:
>>
>>> On Sat, Apr 20, 2013 at 7:59 AM, Mark Nottingham <mnot@mnot.net> wrote:
>>> Agreed, except a new PATCH format that's range-friendly would be necessary. That's not a huge undertaking, because it could reuse at least some of the existing syntax.
>>>   
>>> IMO the simplest solution would be an "Offset" header that simply gives the start offset where the data should be applied. The end offset is implicit through the message length.
>> How does the server know when the whole thing has been uploaded?  Wouldn't we need some kind of "total-length" header, perhaps in the initial PUT?
>>
>> Or are we assuming that the initial PUT is trying to upload everything, and only failure leads to sending the PATCH?
> One possibility would be to have a flag in the request (header or body) that indicates that after this PATCH, it's complete. E.g.,
>
>
> PUT /foo HTTP/1.1  # create the new resource
>
> PATCH /foo HTTP/1.1  # start uploading it...
> PATCH /foo HTTP/1.1  # continue...
> ...
>
> PATCH /foo HTTP/1.1
> Upload-Complete: 1  # we're done!
>
> You'd need conditionals (probably, if-match) on each PATCH to make sure it hasn't changed since the last PATCH (or PUT).
>
> The PUT give the server an opportunity to refuse the request, and its response can advertise the details of how to PATCH it.
>
> The only difficult part I can see is figuring out what to return to a GET that occurs before the upload is complete. And maybe how long to keep these partially-uploaded resources around (although that's really a server implementation detail).

Not too difficult. The processing model for these type of systems falls 
into two categories of locked or lockless. The GEt on locked must be 
responded with some form of error-because-locked response, and GET on 
lockless must always return the current representation with recipient 
system able to cope with corrupted/incomplete representations on its own 
outside of the delivery protocol.

Amos