Re: Resumable Uploads

Mark Nottingham <mnot@mnot.net> Sun, 21 April 2013 02:09 UTC

Return-Path: <ietf-http-wg-request@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0DE0B21F86FA for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sat, 20 Apr 2013 19:09:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.443
X-Spam-Level:
X-Spam-Status: No, score=-10.443 tagged_above=-999 required=5 tests=[AWL=0.156, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p1EirEpfpd6K for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sat, 20 Apr 2013 19:09:39 -0700 (PDT)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) by ietfa.amsl.com (Postfix) with ESMTP id 3EB3421F86C3 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Sat, 20 Apr 2013 19:09:39 -0700 (PDT)
Received: from lists by frink.w3.org with local (Exim 4.72) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1UTjiH-0004qq-N1 for ietf-http-wg-dist@listhub.w3.org; Sun, 21 Apr 2013 02:09:09 +0000
Resent-Date: Sun, 21 Apr 2013 02:09:09 +0000
Resent-Message-Id: <E1UTjiH-0004qq-N1@frink.w3.org>
Received: from lisa.w3.org ([128.30.52.41]) by frink.w3.org with esmtp (Exim 4.72) (envelope-from <mnot@mnot.net>) id 1UTjiE-0004pv-65 for ietf-http-wg@listhub.w3.org; Sun, 21 Apr 2013 02:09:06 +0000
Received: from mxout-08.mxes.net ([216.86.168.183]) by lisa.w3.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from <mnot@mnot.net>) id 1UTjiD-0005m7-BE for ietf-http-wg@w3.org; Sun, 21 Apr 2013 02:09:06 +0000
Received: from mnot-mini.mnot.net (unknown [118.209.190.66]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id DD994509B5; Sat, 20 Apr 2013 22:08:42 -0400 (EDT)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))
From: Mark Nottingham <mnot@mnot.net>
In-Reply-To: <2FD45C0A-1A7C-483B-B8E4-5F300B3EDA66@checkpoint.com>
Date: Sun, 21 Apr 2013 12:08:39 +1000
Cc: =?iso-8859-1?Q?Felix_Geisend=F6rfer?= <felix@transloadit.com>, HTTP Working Group <ietf-http-wg@w3.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net>
References: <CADZbJ9dYFGyrceh03M3B0KdKto7160Dis_geh9um0BhVe1re0g@mail.gmail.com> <EA846138-6537-4709-AC44-149873716E29@mnot.net> <CADZbJ9f4wtaFQEsM_wQn-GaTz+fTKZNyfQk6hXG5OL=Lpkhpcw@mail.gmail.com> <2FD45C0A-1A7C-483B-B8E4-5F300B3EDA66@checkpoint.com>
To: Yoav Nir <ynir@checkpoint.com>
X-Mailer: Apple Mail (2.1503)
Received-SPF: pass client-ip=216.86.168.183; envelope-from=mnot@mnot.net; helo=mxout-08.mxes.net
X-W3C-Hub-Spam-Status: No, score=-3.3
X-W3C-Hub-Spam-Report: AWL=-3.282, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001
X-W3C-Scan-Sig: lisa.w3.org 1UTjiD-0005m7-BE 21c1e43b95fa1f4a528b7eaf0024d21e
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Resumable Uploads
Archived-At: <http://www.w3.org/mid/B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/17442
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On 21/04/2013, at 5:43 AM, Yoav Nir <ynir@checkpoint.com> wrote:

> 
> On Apr 20, 2013, at 11:30 AM, Felix Geisendörfer <felix@transloadit.com> wrote:
> 
>> On Sat, Apr 20, 2013 at 7:59 AM, Mark Nottingham <mnot@mnot.net> wrote:
>> Agreed, except a new PATCH format that's range-friendly would be necessary. That's not a huge undertaking, because it could reuse at least some of the existing syntax.
>>  
>> IMO the simplest solution would be an "Offset" header that simply gives the start offset where the data should be applied. The end offset is implicit through the message length.
> 
> How does the server know when the whole thing has been uploaded?  Wouldn't we need some kind of "total-length" header, perhaps in the initial PUT?
> 
> Or are we assuming that the initial PUT is trying to upload everything, and only failure leads to sending the PATCH?

One possibility would be to have a flag in the request (header or body) that indicates that after this PATCH, it's complete. E.g.,


PUT /foo HTTP/1.1  # create the new resource

PATCH /foo HTTP/1.1  # start uploading it...
PATCH /foo HTTP/1.1  # continue...
...

PATCH /foo HTTP/1.1
Upload-Complete: 1  # we're done!

You'd need conditionals (probably, if-match) on each PATCH to make sure it hasn't changed since the last PATCH (or PUT).

The PUT give the server an opportunity to refuse the request, and its response can advertise the details of how to PATCH it.

The only difficult part I can see is figuring out what to return to a GET that occurs before the upload is complete. And maybe how long to keep these partially-uploaded resources around (although that's really a server implementation detail).

It'd be nice if the PATCH format could modify headers cleanly too.

Cheers,

--
Mark Nottingham   http://www.mnot.net/