Re: Draft v1 Update for Resumable Uploads

Guoye Zhang <guoye_zhang@apple.com> Tue, 21 June 2022 06:31 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3459CC15D48E for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Mon, 20 Jun 2022 23:31:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.738
X-Spam-Level:
X-Spam-Status: No, score=-2.738 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_HELO_TEMPERROR=0.01, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NL0eHA9U-4q4 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Mon, 20 Jun 2022 23:30:37 -0700 (PDT)
Received: from lyra.w3.org (lyra.w3.org [128.30.52.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B3B57C15D482 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Mon, 20 Jun 2022 23:30:02 -0700 (PDT)
Received: from lists by lyra.w3.org with local (Exim 4.92) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1o3XLH-0001Ds-QX for ietf-http-wg-dist@listhub.w3.org; Tue, 21 Jun 2022 06:26:27 +0000
Resent-Date: Tue, 21 Jun 2022 06:26:27 +0000
Resent-Message-Id: <E1o3XLH-0001Ds-QX@lyra.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by lyra.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <guoye_zhang@apple.com>) id 1o3XLG-0001Cz-5b for ietf-http-wg@listhub.w3.org; Tue, 21 Jun 2022 06:26:26 +0000
Received: from ma1-aaemail-dr-lapp02.apple.com ([17.171.2.68]) by mimas.w3.org with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from <guoye_zhang@apple.com>) id 1o3XLE-001onI-U0 for ietf-http-wg@w3.org; Tue, 21 Jun 2022 06:26:25 +0000
Received: from pps.filterd (ma1-aaemail-dr-lapp02.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp02.apple.com (8.16.0.42/8.16.0.42) with SMTP id 25L6IU4s062009; Mon, 20 Jun 2022 23:26:11 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=RXME91xNoC1AfBtK3ohJUeGVDFh4kaPYwPNjghDu7RI=; b=JAiXBisMmZTRSP8J6Ij5ksnWbGwHkzCdE5It0djauRgIay8LuU0bhA+5BSWpImreSwRc MKUFzHVa3/ogLEuEyoakQuvql6vjDTCyN1K61wwwv95pDXKP07He3xKufD8C8KnGusmu as2j0ypvzb9cTEDkAe2bgXhq9bdlnkTBmeiwK02yawiz9GMBebXarCdYqR1HtCEe/r9/ DneX7AlEJlFJvtb5pWO8NQBmOq8a4BK5Hlgxmfu6K15ymR6znPCew2H0I+FIzssQx4h3 xb1+myo4aZp2gYeKqJ1NIPW3KTWRUg9/5uQM7LArxq+gpqcNdd8e4WfMVNXVtV0aAh4m WQ==
Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma1-aaemail-dr-lapp02.apple.com with ESMTP id 3gsbct4s7d-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 20 Jun 2022 23:26:11 -0700
Received: from rn-mailsvcp-mmp-lapp01.rno.apple.com (rn-mailsvcp-mmp-lapp01.rno.apple.com [17.179.253.14]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.18.20220407 64bit (built Apr 7 2022)) with ESMTPS id <0RDT0086BDVN5IA0@rn-mailsvcp-mta-lapp02.rno.apple.com>; Mon, 20 Jun 2022 23:26:11 -0700 (PDT)
Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp01.rno.apple.com by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.18.20220407 64bit (built Apr 7 2022)) id <0RDT00T00DEQKK00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Mon, 20 Jun 2022 23:26:11 -0700 (PDT)
X-Va-A:
X-Va-T-CD: ad1c7db98facf33010cfd2d0b48e1bee
X-Va-E-CD: 7fa589823f194c8498e6df6440bddbf3
X-Va-R-CD: 87a202228b76ae5a02807a21fbbc1b7c
X-Va-CD: 0
X-Va-ID: 9a7e53ec-d08e-42bd-9e35-ee9ad882087e
X-V-A:
X-V-T-CD: ad1c7db98facf33010cfd2d0b48e1bee
X-V-E-CD: 7fa589823f194c8498e6df6440bddbf3
X-V-R-CD: 87a202228b76ae5a02807a21fbbc1b7c
X-V-CD: 0
X-V-ID: ab5cde8c-4485-4bb5-bef9-d9423d3b8825
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-21_02:2022-06-17,2022-06-21 signatures=0
Received: from smtpclient.apple (unknown [17.11.144.143]) by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.18.20220407 64bit (built Apr 7 2022)) with ESMTPSA id <0RDT00RH5DVMNW00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Mon, 20 Jun 2022 23:26:10 -0700 (PDT)
From: Guoye Zhang <guoye_zhang@apple.com>
Message-id: <3B21D19E-A95C-4A68-B40F-23A59C9BA5DB@apple.com>
Content-type: multipart/alternative; boundary="Apple-Mail=_FBD94EA0-332A-4A3F-9869-35DC6A02CE27"
MIME-version: 1.0 (Mac OS X Mail 16.0 \(3724.0.1.1.31\))
Date: Mon, 20 Jun 2022 23:26:00 -0700
In-reply-to: <CAAPGdfEgMF6PQjejUP-r2+vT19fS8my4Mu=V_XvwCUy=8r9Abw@mail.gmail.com>
Cc: ietf-http-wg@w3.org
To: Greg Wilkins <gregw@webtide.com>
References: <BED5A5BC-3F7F-47E2-815E-DC0483328DFD@apple.com> <CAAPGdfEgMF6PQjejUP-r2+vT19fS8my4Mu=V_XvwCUy=8r9Abw@mail.gmail.com>
X-Mailer: Apple Mail (2.3724.0.1.1.31)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-21_02:2022-06-17,2022-06-21 signatures=0
Received-SPF: pass client-ip=17.171.2.68; envelope-from=guoye_zhang@apple.com; helo=ma1-aaemail-dr-lapp02.apple.com
X-W3C-Hub-DKIM-Status: validation passed: (address=guoye_zhang@apple.com domain=apple.com), signature is good
X-W3C-Hub-Spam-Status: No, score=-9.7
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.597, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1o3XLE-001onI-U0 65313d0c86a9bd21e35771dd6ede56e2
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Draft v1 Update for Resumable Uploads
Archived-At: <https://www.w3.org/mid/3B21D19E-A95C-4A68-B40F-23A59C9BA5DB@apple.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/40195
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>


> On Jun 20, 2022, at 22:42, Greg Wilkins <gregw@webtide.com> wrote:
> 
> Gouye,
> 
> It was not immediately clear to me on initially reading the draft exactly what the intent of this proposal is with regards to the existing usage of multipart by clients to upload files (https://datatracker.ietf.org/doc/html/rfc7578)  Some more text in the intro (and 3.2 series of parts) to help to make the intent clearer.
> 
> To be clear myself, I understand that:
> This is not a replacement for multipart uploads, rather it is a mechanism that could be applied on top that could allow a multipart upload (among other content types) to be interrupted and resumed.
> The usage of this mechanism to deliberately split an upload into parts is distinct from RFC7578 parts (although no reason why each RFC7578 part could not be sent in a separate resumable request if server support is known).
Yes, you are entirely correct. tus-v2 supports multipart upload, and the parts in tus-v2 has no relationship with parts in multipart. The multipart body is treated as an opaque binary blob like any other request bodies.
> The use of this mechanism to initiate parallel requests to "speed up" the upload is explicitly not allowed.  Perhaps you should note that this is asymmetric to down loads with range request, which can be done in parallel.  
Good point! We will add a note about it.

We think parallel upload places unnecessary burden on the server code to support it, and it is generally an abuse of the network. It is not supported by the current production tus protocol, either.
> When the server is queried for an offset, it is free to pick an offset as it likes. For example, it could wind back the transfer to the last boundary of a multi-part content, even if some of those bytes had already been received.      Could the server use this mechanism to skip ahead?  for example during a large upload the server determines that it is only interested in the last part of the content. The if the server closes the current transfer and when a offset query request is received, can it send an offset in advance of any bytes sent by the client, so the client would then need to skip through to the offset (rather than rewind)? 
This is a really interesting use case. We do not prevent this in the current draft, and it will probably just work with our current experimental implementations. Is there a concrete example where this capability is useful?

We’ll definitely discuss it internally. If it isn’t too much trouble for client implementations, I don’t see any reason not to support it explicitly in the draft.

Guoye
> 
> regards
> 
> 
> 
> 
> On Fri, 17 Jun 2022 at 07:36, Guoye Zhang <guoye_zhang@apple.com <mailto:guoye_zhang@apple.com>> wrote:
>> Hi all,
>> 
>> Our previous resumable upload draft generated a lot of discussions. I’m glad to announce that we have a new draft ready to address many feedbacks that suggested adopting the PATCH method. In this draft, we split the Upload Transfer Procedure into 2 separate procedures: Upload Creation Procedure and Upload Appending Procedure.
>> 
>> https://datatracker.ietf.org/doc/draft-tus-httpbis-resumable-uploads-protocol/
>> 
>> 1. Content-Range
>> 
>> We attempted adopting Content-Range header, however, we realized that it doesn’t support unknown lengths which is an important use case that our clients require. Therefore we kept Upload-Offset and Upload-Incomplete headers.
>> 
>> We are open to discuss other options, such as modifying the semantics of the Content-Range header if that’s preferred, although it might cause more breakages than defining new headers.
>> 
>> 2. Media types
>> 
>> PATCH currently doesn’t define a media type. We went through the list of media types but couldn’t find the appropriate category for the Upload Appending Procedure. It is a generic byte-appending operation that can modify any types of media, so we don’t think it fits into an application media type.
>> 
>> We are open to suggestions if a media type is desired.
>> 
>> 3. 1xx intermediate response
>> 
>> We surveyed the most popular HTTP libraries in many languages, and nearly all of them consider 1xx responses an internal signaling mechanism so they don’t expose the ability for applications to handle them. (We are also guilty of this as maintainers of URLSession API on Apple platforms.) If we use 1xx response for any critical information, it would prevent nearly all tus-v1 adopters to switch to this new protocol until it’s natively supported in HTTP libraries.
>> 
>> We think having just the feature detection part using 1xx response is a good balance, both eliminating any extra round trips for HTTP libraries implementing this protocol and allowing application adopters to ignore it.
>> 
>> 4. Can we PATCH a PATCH?
>> 
>> Yes, Upload Creation Procedure supports any method, including PATCH. We included a section “Request Identification” about the nuances in this area. Unfortunately, this added complexity is the result of splitting the procedures, but we don’t think it will complicate the implementations in most cases. Servers can still decide what methods make sense for their use case and whether to support PATCH.
>> 
>> 
>> Looking forward to continuing the discussions and refinements of the draft.
>> 
>> Best regards,
>> Guoye Zhang
> 
> 
> -- 
> Greg Wilkins <gregw@webtide.com <mailto:gregw@webtide.com>> CTO http://webtide.com <http://webtide.com/>