Re: Resumable Uploads

Carsten Bormann <cabo@tzi.org> Sun, 21 April 2013 11:25 UTC

Return-Path: <ietf-http-wg-request@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B781F21F8E96 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sun, 21 Apr 2013 04:25:36 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -8.93
X-Spam-Level:
X-Spam-Status: No, score=-8.93 tagged_above=-999 required=5 tests=[AWL=1.669, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CNXjpvla-hLm for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sun, 21 Apr 2013 04:25:36 -0700 (PDT)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) by ietfa.amsl.com (Postfix) with ESMTP id 1BAC821F8E04 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Sun, 21 Apr 2013 04:25:35 -0700 (PDT)
Received: from lists by frink.w3.org with local (Exim 4.72) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1UTsNW-0005Z7-SU for ietf-http-wg-dist@listhub.w3.org; Sun, 21 Apr 2013 11:24:18 +0000
Resent-Date: Sun, 21 Apr 2013 11:24:18 +0000
Resent-Message-Id: <E1UTsNW-0005Z7-SU@frink.w3.org>
Received: from maggie.w3.org ([128.30.52.39]) by frink.w3.org with esmtp (Exim 4.72) (envelope-from <cabo@tzi.org>) id 1UTsNT-0005YO-9z for ietf-http-wg@listhub.w3.org; Sun, 21 Apr 2013 11:24:15 +0000
Received: from mailhost.informatik.uni-bremen.de ([134.102.201.18] helo=informatik.uni-bremen.de) by maggie.w3.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from <cabo@tzi.org>) id 1UTsNS-0007Hv-1d for ietf-http-wg@w3.org; Sun, 21 Apr 2013 11:24:15 +0000
X-Virus-Scanned: amavisd-new at informatik.uni-bremen.de
Received: from smtp-fb3.informatik.uni-bremen.de (smtp-fb3.informatik.uni-bremen.de [134.102.224.120]) by informatik.uni-bremen.de (8.14.4/8.14.4) with ESMTP id r3LBNeLV007588; Sun, 21 Apr 2013 13:23:40 +0200 (CEST)
Received: from [192.168.217.105] (p54890A03.dip0.t-ipconnect.de [84.137.10.3]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp-fb3.informatik.uni-bremen.de (Postfix) with ESMTPSA id 40C3B39C1; Sun, 21 Apr 2013 13:23:40 +0200 (CEST)
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))
Content-Type: text/plain; charset=iso-8859-1
From: Carsten Bormann <cabo@tzi.org>
In-Reply-To: <B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net>
Date: Sun, 21 Apr 2013 13:23:39 +0200
Cc: Yoav Nir <ynir@checkpoint.com>, =?iso-8859-1?Q?Felix_Geisend=F6rfer?= <felix@transloadit.com>, HTTP Working Group <ietf-http-wg@w3.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <3B78E040-4AEE-4194-8ECA-6F8DF8FED75A@tzi.org>
References: <CADZbJ9dYFGyrceh03M3B0KdKto7160Dis_geh9um0BhVe1re0g@mail.gmail.com> <EA846138-6537-4709-AC44-149873716E29@mnot.net> <CADZbJ9f4wtaFQEsM_wQn-GaTz+fTKZNyfQk6hXG5OL=Lpkhpcw@mail.gmail.com> <2FD45C0A-1A7C-483B-B8E4-5F300B3EDA66@checkpoint.com> <B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net>
To: Mark Nottingham <mnot@mnot.net>
X-Mailer: Apple Mail (2.1503)
Received-SPF: none client-ip=134.102.201.18; envelope-from=cabo@tzi.org; helo=informatik.uni-bremen.de
X-W3C-Hub-Spam-Status: No, score=-5.6
X-W3C-Hub-Spam-Report: AWL=-1.350, BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001
X-W3C-Scan-Sig: maggie.w3.org 1UTsNS-0007Hv-1d c76f082288f5fc2c08678dc1bc53910d
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Resumable Uploads
Archived-At: <http://www.w3.org/mid/3B78E040-4AEE-4194-8ECA-6F8DF8FED75A@tzi.org>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/17447
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On Apr 21, 2013, at 04:08, Mark Nottingham <mnot@mnot.net> wrote:

> PUT /foo HTTP/1.1  # create the new resource
> 
> PATCH /foo HTTP/1.1  # start uploading it...
> PATCH /foo HTTP/1.1  # continue...
> ...
> 
> PATCH /foo HTTP/1.1
> Upload-Complete: 1  # we're done!

This is starting to look like draft-ietf-core-block, and is bound to have a number of the same characteristics (except that CoAP blocks are small and fixed-size).
(Of course, -block is solving a different problem...)

How to do this properly probably depends on whether the new resource is freestanding, is bound to be part of a collection, or is an update of an existing resource.

The first PUT may not really create the resource, it might create a temporary that can become the new representation of the resource.
The resource representation is only really completely uploaded with the final patch.

If access to the resource-in-creation is exclusive (e.g., a new URI that is only used by the uploader), the resource can grow in place.
If you don't want to indicate explicitly that it is done, the client can do that implicitly by just starting to use the URI.
Garbage collection (incompletely uploaded resources) is more fun, though.

If access is non-exclusive (multiple clients could be updating it in parallel), you can lock, or you can create an explicit temporary for each uploader, enabling "the last update wins".

Since the original project seems to be about uploading videos, exclusive access is probably the better paradigm.
We don't know if a PATCH (or even the initial PUT) is going to go through completely, so indicating at the start that it is the last one may be impossible.
An explicit close, conditional on the size actually received, may be the best way to do this.
This could use the LINK method or something else.

Grüße, Carsten