Re: Resumable Uploads

Felix Geisendörfer <felix@transloadit.com> Sun, 21 April 2013 10:15 UTC

Return-Path: <ietf-http-wg-request@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DBA7421F8EBE for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sun, 21 Apr 2013 03:15:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.376
X-Spam-Level:
X-Spam-Status: No, score=-9.376 tagged_above=-999 required=5 tests=[AWL=0.300, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BhN9ZZIf4qQv for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sun, 21 Apr 2013 03:15:24 -0700 (PDT)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) by ietfa.amsl.com (Postfix) with ESMTP id 9A35021F8E4B for <httpbisa-archive-bis2Juki@lists.ietf.org>; Sun, 21 Apr 2013 03:15:24 -0700 (PDT)
Received: from lists by frink.w3.org with local (Exim 4.72) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1UTrH9-0003cP-2B for ietf-http-wg-dist@listhub.w3.org; Sun, 21 Apr 2013 10:13:39 +0000
Resent-Date: Sun, 21 Apr 2013 10:13:39 +0000
Resent-Message-Id: <E1UTrH9-0003cP-2B@frink.w3.org>
Received: from maggie.w3.org ([128.30.52.39]) by frink.w3.org with esmtp (Exim 4.72) (envelope-from <felix.geisendoerfer@transloadit.com>) id 1UTrH3-0003bg-Ge for ietf-http-wg@listhub.w3.org; Sun, 21 Apr 2013 10:13:33 +0000
Received: from mail-vc0-f175.google.com ([209.85.220.175]) by maggie.w3.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.72) (envelope-from <felix.geisendoerfer@transloadit.com>) id 1UTrH2-0004Gj-HU for ietf-http-wg@w3.org; Sun, 21 Apr 2013 10:13:33 +0000
Received: by mail-vc0-f175.google.com with SMTP id lf11so3014221vcb.6 for <ietf-http-wg@w3.org>; Sun, 21 Apr 2013 03:13:06 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:sender:x-originating-ip:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:x-gm-message-state; bh=SbawEWmmFE61cTtxu1Kc7MtjLNuWM7Qj4io7wh6E4Vk=; b=gZ9XoEQqZW5RfmfdjCVC+w3T3k90dr98v2XT3yUlZRjz+E3kGoZyotNWkBpz2yKJ0B YkXve1g0QK9jeKsCdjvQON1x7UNnIlzPqSd2Bicv12oQGjN/ATzZUWDM41bSojQp0MgR Xab0tHtbiy4Lc5JHquZN97vUDF2H2XJnPreTlGBBYdWV66CTz/W4PRyO8VEvNBU/Pmd8 vUqr5tExUFlp+cnTU0o+Ufi8+uAEASGUiOKYNz9tYCoIlvc+m1lY4g1Af6n24Mn4z39O DZ9Fx0JIpVn06zuf+mTGgIX1YFOuWRCxm8Jtmq8DMeDWDNDWD7r1/OvuT0kXfNXspFFB 2T1g==
X-Received: by 10.52.103.50 with SMTP id ft18mr5842496vdb.108.1366539186707; Sun, 21 Apr 2013 03:13:06 -0700 (PDT)
MIME-Version: 1.0
Sender: felix.geisendoerfer@transloadit.com
Received: by 10.58.15.165 with HTTP; Sun, 21 Apr 2013 03:12:45 -0700 (PDT)
X-Originating-IP: [91.64.81.5]
In-Reply-To: <51735224.5010405@treenet.co.nz>
References: <CADZbJ9dYFGyrceh03M3B0KdKto7160Dis_geh9um0BhVe1re0g@mail.gmail.com> <EA846138-6537-4709-AC44-149873716E29@mnot.net> <CADZbJ9f4wtaFQEsM_wQn-GaTz+fTKZNyfQk6hXG5OL=Lpkhpcw@mail.gmail.com> <2FD45C0A-1A7C-483B-B8E4-5F300B3EDA66@checkpoint.com> <B025EE2E-D707-4FFF-8FB7-33A7AF18282A@mnot.net> <51735224.5010405@treenet.co.nz>
From: =?ISO-8859-1?Q?Felix_Geisend=F6rfer?= <felix@transloadit.com>
Date: Sun, 21 Apr 2013 12:12:45 +0200
X-Google-Sender-Auth: -46QglEqDQDbXAa3o43WJACh_aE
Message-ID: <CADZbJ9eEeBUYP_5dDXvqd-grY035-QSeyyGZrUR=2VviB6CGCw@mail.gmail.com>
To: Amos Jeffries <squid3@treenet.co.nz>
Cc: HTTP Working Group <ietf-http-wg@w3.org>
Content-Type: multipart/alternative; boundary=047d7bacb7e4cbe1bb04dadc320c
X-Gm-Message-State: ALoCoQmZsE5RkmjN4lS73dR4ICMObhb52gWW0iwgJsf6397sNODK5PfJ6e8ljehA8HWB3ILxqNSR
Received-SPF: none client-ip=209.85.220.175; envelope-from=felix.geisendoerfer@transloadit.com; helo=mail-vc0-f175.google.com
X-W3C-Hub-Spam-Status: No, score=-5.3
X-W3C-Hub-Spam-Report: AWL=-2.687, BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7
X-W3C-Scan-Sig: maggie.w3.org 1UTrH2-0004Gj-HU c12a7beec89ffcecd8017cad1a02dcbc
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Resumable Uploads
Archived-At: <http://www.w3.org/mid/CADZbJ9eEeBUYP_5dDXvqd-grY035-QSeyyGZrUR=2VviB6CGCw@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/17446
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On Sat, Apr 20, 2013 at 11:09 PM, Ilari Liusvaara <
ilari.liusvaara@elisanet.fi> wrote:

> This is because JS code can't read huge chunks of file at once[1].
>

AFAIK that's not true. File API 8.1 says: "The File interface, which
inherits from Blob, is immutable, and thus represents file data that can be
read into memory at the time a read operation is initiated." [1]. In my
testing this optimization is found in all browser implementations.

On Sat, Apr 20, 2013 at 11:20 PM, Daniel Stenberg <daniel@haxx.se> wrote:
>
> I agree with Fredrik's statement that a simple PUT + offset header would
> be easiest for a client (and possibly server too), but I just previously
> quoted the reasoning in the spec as to why it currently doesn't think that
> is a good idea. It would be more productive if we would instead counter
> that argument or present an alternative way.


My proposal is to use PATCH + Offset header. PUT could still be used for
creating the resource initially and setting expectations for final size,
checksum, etc.

Overall I think there are a few separate things that are currently lacking
http primitives. In order of priority:

1. Defining a simple way to insert data at a given offset of a resource.
(PATCH + Offset header seems nice/simple)
2. Defining a fixed file size for a resource when creating it (suggestion:
"Final-Length" header), regardless of how much data will be transmitted
with the creation request.
3. Defining a way for a server to provide the state of a partial resource
via HEAD requests. (could also be a "Offset" header)

(I've left out checksums, EOF/Upload-Complete and a few other things on
this list, just to keep it simple for now.)

These primitives could then be combined into various client/server patterns
depending on application requirements.

I don't think the http spec should define the server behavior of GET
requests on partially uploaded resources, as there are many patterns that
would make sense for different applications.

[1]: http://www.w3.org/TR/FileAPI/#file-attrs

--
Felix Geisendörfer (felixge.de)
Co-Founder, Transloadit (transloadit.com)