Re: Resumable Uploads

Felix Geisendörfer <felix@transloadit.com> Sat, 20 April 2013 08:31 UTC

Return-Path: <ietf-http-wg-request@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 60D7221F9265 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sat, 20 Apr 2013 01:31:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.076
X-Spam-Level:
X-Spam-Status: No, score=-9.076 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, J_CHICKENPOX_32=0.6, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BalxDxZY+IFp for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sat, 20 Apr 2013 01:31:40 -0700 (PDT)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) by ietfa.amsl.com (Postfix) with ESMTP id A8DAD21F9298 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Sat, 20 Apr 2013 01:31:40 -0700 (PDT)
Received: from lists by frink.w3.org with local (Exim 4.72) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1UTTCd-0006ZP-Ju for ietf-http-wg-dist@listhub.w3.org; Sat, 20 Apr 2013 08:31:23 +0000
Resent-Date: Sat, 20 Apr 2013 08:31:23 +0000
Resent-Message-Id: <E1UTTCd-0006ZP-Ju@frink.w3.org>
Received: from maggie.w3.org ([128.30.52.39]) by frink.w3.org with esmtp (Exim 4.72) (envelope-from <felix.geisendoerfer@transloadit.com>) id 1UTTCZ-0006Ya-MO for ietf-http-wg@listhub.w3.org; Sat, 20 Apr 2013 08:31:19 +0000
Received: from mail-vc0-f179.google.com ([209.85.220.179]) by maggie.w3.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.72) (envelope-from <felix.geisendoerfer@transloadit.com>) id 1UTTCZ-0003OT-0U for ietf-http-wg@w3.org; Sat, 20 Apr 2013 08:31:19 +0000
Received: by mail-vc0-f179.google.com with SMTP id hz11so4719888vcb.10 for <ietf-http-wg@w3.org>; Sat, 20 Apr 2013 01:30:53 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:sender:x-originating-ip:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:x-gm-message-state; bh=iYCqUbeP9uuOViadHCx1KK59c1R06KQKXVfS9lLAedI=; b=VuUCh2a08fpM9UqrArxVT83LDYGcx1HjXhEaBJXMAOq2fsQlMUhq0oCFNawNte0MC7 w6M2YqQNtlqbUkS47ci+vYF/iNAx3e/zZu3SIM8U2VI2hFWtPAk0mlJr37r6wh4Q17Jg AJfw7k1XxFVJo2KBcbChZtyTMC/wnBSb7A7hQlg7E/6tGqk5LTQKv3CSTox407dzO1gD SyBv4v3xEL2SgQ6ol1m9nJhF0/1aA8nfXjuzgilCidzfdL0pYGYs5w9Sv0A9PTtfQJeY aNS30OcXaPodRhfoxPBisZKdgTiyj0+9wyLqSuQ3GmblInOk3/gHYsUpy03Q2XzmJVRY h64w==
X-Received: by 10.220.223.202 with SMTP id il10mr14075192vcb.4.1366446653321; Sat, 20 Apr 2013 01:30:53 -0700 (PDT)
MIME-Version: 1.0
Sender: felix.geisendoerfer@transloadit.com
Received: by 10.58.15.165 with HTTP; Sat, 20 Apr 2013 01:30:33 -0700 (PDT)
X-Originating-IP: [91.64.81.5]
In-Reply-To: <EA846138-6537-4709-AC44-149873716E29@mnot.net>
References: <CADZbJ9dYFGyrceh03M3B0KdKto7160Dis_geh9um0BhVe1re0g@mail.gmail.com> <EA846138-6537-4709-AC44-149873716E29@mnot.net>
From: Felix Geisendörfer <felix@transloadit.com>
Date: Sat, 20 Apr 2013 10:30:33 +0200
X-Google-Sender-Auth: qxsHPbTS4c9Bo6kjd7Di8t09WmE
Message-ID: <CADZbJ9f4wtaFQEsM_wQn-GaTz+fTKZNyfQk6hXG5OL=Lpkhpcw@mail.gmail.com>
To: Mark Nottingham <mnot@mnot.net>
Cc: HTTP Working Group <ietf-http-wg@w3.org>
Content-Type: multipart/alternative; boundary="14dae9cdc487607b2b04dac6a7ad"
X-Gm-Message-State: ALoCoQk1E8aosPr9JltoqmG8HKDFDRquR83UW6N5F8iu+FuEr4qPgs9T3Gz5H/f/J9iVcmKnPV/x
Received-SPF: none client-ip=209.85.220.179; envelope-from=felix.geisendoerfer@transloadit.com; helo=mail-vc0-f179.google.com
X-W3C-Hub-Spam-Status: No, score=-4.9
X-W3C-Hub-Spam-Report: AWL=-4.173, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7
X-W3C-Scan-Sig: maggie.w3.org 1UTTCZ-0003OT-0U 59c77b4015c9efb741679ca445c5a4a8
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Resumable Uploads
Archived-At: <http://www.w3.org/mid/CADZbJ9f4wtaFQEsM_wQn-GaTz+fTKZNyfQk6hXG5OL=Lpkhpcw@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/17410
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On Sat, Apr 20, 2013 at 7:59 AM, Mark Nottingham <mnot@mnot.net> wrote:

> Agreed, except a new PATCH format that's range-friendly would be
> necessary. That's not a huge undertaking, because it could reuse at least
> some of the existing syntax.
>

IMO the simplest solution would be an "Offset" header that simply gives the
start offset where the data should be applied. The end offset is implicit
through the message length.


> I'd be willing to help work on this, or just provide input / reviews.
>

That's fantastic, thank you so much! I'm also very interested in helping in
whatever way possible.

On Fri, Apr 19, 2013 at 10:08 PM, Carsten Bormann <cabo@tzi.org> wrote:

Your assumption seems to be that if an upload breaks, it breaks cleanly,
> i.e. all data that have been received by the server are fine up to the last
> byte.
>

Yes, but IMO we should assume that the underlaying transport layer is not
corrupting data under normal circumstances.

Applications that require stronger guarantess should split their uploads
into small chunk requests with individual checksums and instruct their
servers to not process anything that doesn't match the checksum. This
carries a higher overhead, but such is life : ).

Our goal for tus.io is to describe the simplest interoperable resumable
upload strategy over http that will work in most situations (the core). The
rest of the protocol will cover extensions (chunking, checksums, uploading
chunks in parallel, etc.) which clients / servers can choose to implement
as well. Ideally a nice ecosystem of compatible libraries will then
flourish on top.

Cheers,
--
Felix Geisendörfer (felixge.de)
Co-Founder, Transloadit (transloadit.com)