Re: Resumable Upload draft updates
Marius Kleidl <marius@transloadit.com> Thu, 25 July 2024 11:35 UTC
Received: by ietfa.amsl.com (Postfix) id 50EA2C1CAF3B; Thu, 25 Jul 2024 04:35:17 -0700 (PDT)
Delivered-To: ietfarch-httpbisa-archive-bis2juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 503D7C180B49 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Thu, 25 Jul 2024 04:35:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -7.856
X-Spam-Level:
X-Spam-Status: No, score=-7.856 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=w3.org header.b="C1b5BpUs"; dkim=pass (2048-bit key) header.d=w3.org header.b="MiTcLpXK"; dkim=pass (1024-bit key) header.d=transloadit.com header.b="JniUlzfy"
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zyefQ8LQVM1J for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Thu, 25 Jul 2024 04:35:13 -0700 (PDT)
Received: from mab.w3.org (mab.w3.org [IPv6:2600:1f18:7d7a:2700:d091:4b25:8566:8113]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1221AC180B4E for <httpbisa-archive-bis2Juki@ietf.org>; Thu, 25 Jul 2024 04:35:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Subject:Content-Type:Cc:To:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To; bh=4/YO/uWYaz5/QMuBdKw7/YqK7gMB4zyRKQEZXF9M4QE=; b=C1b5BpUspVtmCPKtwBUIwPDJvL a43wt+VJKD2Q4/gAbG8kFvALerz4oNAsBs4F2GnvMShFTyVCYzmE9rjf6FFIoUn6d2Y7lccoEjrKj xx3rhOZyy3Q3PJj4bOU3E5wGp3VFmP2q5TvvAebgTrxVkTXrouJ/AU0Ah3O9S8ntGKjgVOLDibcXA ZVaEiMt2g8RuKoOlAmynwqeFcL0cTZut8zFIBUSl2+BLEXAGKThSUAhDZdfeGcTCoqx6cT8YQXv1w seUMevJA6+jiHmeM00Fa9BITxxmTN/IIq2Ff3eqeD3Kt7SOqVTdaVqMa04WBLldEYt40tpVqCjAPh gYQoLOmg==;
Received: from lists by mab.w3.org with local (Exim 4.96) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1sWwk8-006rqd-1C for ietf-http-wg-dist@listhub.w3.org; Thu, 25 Jul 2024 11:34:44 +0000
Resent-Date: Thu, 25 Jul 2024 11:34:44 +0000
Resent-Message-Id: <E1sWwk8-006rqd-1C@mab.w3.org>
Received: from ip-10-0-0-224.ec2.internal ([10.0.0.224] helo=puck.w3.org) by mab.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <marius@transloadit.com>) id 1sWwk6-006rpi-0x for ietf-http-wg@listhub.w3.internal; Thu, 25 Jul 2024 11:34:42 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=w3.org; s=s1; h=Content-Type:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To; bh=4/YO/uWYaz5/QMuBdKw7/YqK7gMB4zyRKQEZXF9M4QE=; t=1721907282; x=1722771282; b=MiTcLpXK2YE2fWyfmDXbYeBRnOqkba/4X7V4Dqp9b6jj3W+nR1LMmnIDQYGSJQLfHQKBYqDHsOc 8K/ZfC4aiQRCejWn6LsGZ17cVUG9Ubm0qSWfmgu4P7vaYZGK67hsnPnh0MypLgGiUQ3VjYEzKyhHQ 658IVMmeklDkpDHZu2xPc/WkI4s05gyTiV0zR3+BRgN5nVz3/5HRStaC+NO4NGBmiNEndJ+AKND0g XR3gqhkcUh3pX6E8udV+t6o7wHPXrCabl2t4Kqx85IiF9fVDqfvqQHReq5tf6acAdNd2zEch9Vb1+ 46IdOkYJBBOzPiUou8tcbP6Pz54FKj3QRjSQ==;
Received-SPF: pass (puck.w3.org: domain of transloadit.com designates 2a00:1450:4864:20::331 as permitted sender) client-ip=2a00:1450:4864:20::331; envelope-from=marius@transloadit.com; helo=mail-wm1-x331.google.com;
Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by puck.w3.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from <marius@transloadit.com>) id 1sWwk5-004PDt-1k for ietf-http-wg@w3.org; Thu, 25 Jul 2024 11:34:42 +0000
Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-4266b1f1b21so6160285e9.1 for <ietf-http-wg@w3.org>; Thu, 25 Jul 2024 04:34:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=transloadit.com; s=google; t=1721907277; x=1722512077; darn=w3.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=4/YO/uWYaz5/QMuBdKw7/YqK7gMB4zyRKQEZXF9M4QE=; b=JniUlzfycbzJzngWolEaZgHuEmNobmNAUImdn+Z8OOr8MDUjyXy0tc7Gh+pRk7hzGW kkQGI3DgABypnbnbIueaQeodax/npGtKa+1ltoz3lyPYnR7ShAPqJVf4mJGym3RMqliu RAX3MknDEFew/0i5cm+UynMo9diWwvzMdn6M4=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721907277; x=1722512077; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4/YO/uWYaz5/QMuBdKw7/YqK7gMB4zyRKQEZXF9M4QE=; b=l9tmMEiCBKuv5sJmuxzRUveDM3AexfkReEMJ2w4jsh1xHggJMcwB5L4AMx8PZLJmBa O/6KfYZyaHisNuZ2ZAkZv9DImgFpDQPD19dOyOrNYjsyjg8qm348b1H1PC2qgblpnpAW io7IhlTSQJVAQajoCHOy0Ze77tWsM01snSsKGSzqbaQiAmAIld9+nhNtUKAryj+J3Uue 6/Y7QckCPGHOKGt3U3OP+c/sw97HnSFhSPJhJN8qKMvZfvsazq2+LI9l/5Duffk07Mxg XH5AC1Ups5LWrtnCeb9b7mylWWnD2NhNYuK8Kh+iB9JwLI4knn969unRhIELrp1RLu9C 6Nfg==
X-Gm-Message-State: AOJu0YyAuYRl6M49qyq3pcClA/xFKOycxI7/M58KRXAu+jPn42a8sY9H 42MKwFfCqjE86zxb9yhbZ+UDXslOiS5Ma5ofMcgUhij9REcJhJx29aMNcCq6ldUXaday0GR1mmg yiV9AatdHdIfRmG53sxCsNrM4i/0CpsSvKl2pS2wGby4EefLqb7QzRw==
X-Google-Smtp-Source: AGHT+IFj1Yuo3+fOcFtwp+BqkZhvu3Kj5oxYZco/EDNgCgKO3tIwVU6dQuMI4a8GkwUo7SimN81kvIHv8tiW6AA4sbA=
X-Received: by 2002:a5d:4a4b:0:b0:367:8a2b:7354 with SMTP id ffacd0b85a97d-36b3637e128mr1124121f8f.11.1721907277139; Thu, 25 Jul 2024 04:34:37 -0700 (PDT)
MIME-Version: 1.0
References: <CAChr6Szm_5j1p9QKdTN3mjvEvQ-JwcPT1Y8XeOPqGu25DwBijQ@mail.gmail.com>
In-Reply-To: <CAChr6Szm_5j1p9QKdTN3mjvEvQ-JwcPT1Y8XeOPqGu25DwBijQ@mail.gmail.com>
From: Marius Kleidl <marius@transloadit.com>
Date: Thu, 25 Jul 2024 13:34:26 +0200
Message-ID: <CANY19NsOhrTb6ZSKWyd-0Fo+Ac+1f_vH3y1BxTCCOPPAhYCDcA@mail.gmail.com>
To: Rob Sayre <sayrer@gmail.com>
Cc: "ietf-http-wg@w3.org Group" <ietf-http-wg@w3.org>
Content-Type: multipart/alternative; boundary="0000000000009733ab061e10c7b5"
X-W3C-Hub-DKIM-Status: validation passed: (address=marius@transloadit.com domain=transloadit.com), signature is good
X-W3C-Hub-Spam-Status: No, score=-4.1
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_PASS=-0.001, HTML_MESSAGE=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: puck.w3.org 1sWwk5-004PDt-1k 0883c6b7e9a512272c905cae2861d67c
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Resumable Upload draft updates
Archived-At: <https://www.w3.org/mid/CANY19NsOhrTb6ZSKWyd-0Fo+Ac+1f_vH3y1BxTCCOPPAhYCDcA@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/52136
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/email/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>
Hi Rob, thanks for the feedback! I don't really have any strong opinions, but it would be nice to compare it > to popular storage solutions and see if you really need all of the correct > but maybe too clever HTTP mechanisms. AWS S3 offers multipart uploads (not to be confused with the multipart media type): https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html. The client first creates a multipart upload resource and can then upload individual parts to the resource. Each part (except for the last ones) has a minimum size of 5 MB. Once all parts are uploaded, an additional request is used to commit the upload. Parts from interrupted requests is not saved and must be retransmitted. Google Cloud Storage offers resumable uploads as described in https://cloud.google.com/storage/docs/resumable-uploads. Their approach is very similar to the draft in that a resumable upload resource is created, to which the client can then append data to via individual requests. Data from interrupted requests seems to be saved. A client resumes the upload by querying the uploaded data range. Overall, our methods are very similar, although the used methods and header fields are slightly different. Azure Blob Storage offers Append Blobs, which seem to act similar to AWS S3' multipart uploads: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs But I haven't used them much on my own. I wonder if it's helpful to include such a comparison in the draft. The one thing that stuck out to me was the OPTIONS request, I thought that > might be better in the TLS handshake. I would like to emphasize here again that the OPTIONS request is optional and a client can start an upload without. Putting upload limits into the TLS handshake is interesting but not ideal as upload limits belong to a resource and are not a property of the connection. In addition, getting this information from the TLS handshake on the client-side is not always possible, depending on the HTTP API (e.g. the Fetch API does not expose this). Configuring this on the server-side could also be non-trivial if a TLS-terminating proxy is involved. The draft contains a similar discussion of different methods to indicate server support for resumable uploads: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload-04#name-feature-detection-2. It does not mention TLS handshake, but touches on similar topics. Best regards, Marius On Wed, Jul 24, 2024 at 9:03 PM Rob Sayre <sayrer@gmail.com> wrote: > Hi, > > This is in reference to the IETF 120 session and > > < > https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload-04 > > > > Firstly, people continually reimplement this feature, so it would be nice > to standardize it. I don't really have any strong opinions, but it would be > nice to compare it to popular storage solutions and see if you really need > all of the correct but maybe too clever HTTP mechanisms. The one thing that > stuck out to me was the OPTIONS request, I thought that might be better in > the TLS handshake. > > All of those storage servers have this, but the chunks are often very > large, like 100MB. That works great if you're uploading giant files > on servers. But the other thing is that your favorite social networks > definitely do this in much smaller slices when you upload a photo. This is > so you can take a picture, get on the subway, and it all works in a few > minutes. I guess "small objects in the presence of packet loss" would be > the general term. That is the same design as the storage servers, it just > slices files up a lot more. > > thanks, > Rob > >
- Resumable Upload draft updates Marius Kleidl
- Re: Resumable Upload draft updates Marius Kleidl
- Re: Resumable Upload draft updates Michael Toomim
- Re: Resumable Upload draft updates Michael Toomim
- Re: Resumable Upload draft updates Michael Toomim
- Re: Resumable Upload draft updates Marius Kleidl
- re: Resumable Upload draft updates Rob Sayre
- Re: Resumable Upload draft updates Marius Kleidl
- Re: Resumable Upload draft updates Rob Sayre
- Re: Resumable Upload draft updates Lucas Pardue
- Re: Resumable Upload draft updates Rob Sayre
- Re: Resumable Upload draft updates Rob Sayre