Resent-Date: Tue, 23 Jul 2024 08:08:02 +0000
Resent-Message-Id: <E1sWAZ0-0010WZ-0k@mab.w3.org>
Received-SPF: pass (puck.w3.org: domain of transloadit.com designates 2a00:1450:4864:20::435 as permitted sender) client-ip=2a00:1450:4864:20::435; envelope-from=marius@transloadit.com; helo=mail-wr1-x435.google.com;
MIME-Version: 1.0
References: <CANY19Ns76du=x04xtejdRrdNT2Lq_=5Huo5oH_KQ9qg5bAO89A@mail.gmail.com>
 <29911fc7-cfa3-4ea4-8ee0-822781060fa5@gmail.com>
In-Reply-To: <29911fc7-cfa3-4ea4-8ee0-822781060fa5@gmail.com>
From: Marius Kleidl <marius@transloadit.com>
Date: Tue, 23 Jul 2024 10:07:43 +0200
Message-ID: <CANY19Nt5XSjNguTY2poHvdLQF7p=TAEb1hX1p=ZS2siMCf-U0Q@mail.gmail.com>
To: Michael Toomim <toomim@gmail.com>
Cc: ietf-http-wg@w3.org
Content-Type: multipart/alternative; boundary="000000000000aad24e061de5a887"
Subject: Re: Resumable Upload draft updates
Archived-At: <https://www.w3.org/mid/CANY19Nt5XSjNguTY2poHvdLQF7p=TAEb1hX1p=ZS2siMCf-U0Q@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list

--000000000000aad24e061de5a887
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Michael,

there are indeed similarities in our approaches and I like the perspective
with which you are looking at the problem! At the end of the day, most
approaches to resumable uploads use the same underlying concepts but framed
with different terminology, field names or status codes.

Focusing specifically on the usage of your draft for uploads, I have a few
comments:

1. What is the parent version of a resource that a client is uploading to?
Requests have to set this header to ensure they are appending at the
correct offset, but I am wondering what parent version would be included in
responses to HEAD requests. The example responses in section 3.3 always use
the 0 byte count.
2. Can a client upload from a streaming data source without knowing the
entire file size upfront? Is the including of the Current-Version header
mandatory when starting an upload?
3. The example request for case B in section 3.3.2 includes the
Content-Range header, which appears redundant. Does its value carry
additional information that is not covered by Parents, Current-Version or
Content-Length?
4. Does the method allow the client to split an upload across multiple
requests? In our experience, servers can have an upper limit on the size of
a single request, which requires a client to split an upload and send
multiple consecutive requests to transfer the entire file.

Feel free to ignore them if you they fall outside of the draft's intended
scope. I am looking forward to your presentation on Wednesday!

Best regards,
Marius Kleidl

On Tue, Jul 23, 2024 at 12:51=E2=80=AFAM Michael Toomim <toomim@gmail.com> =
wrote:

> It was great seeing this progress on resumeable uploads in HTTPWG today!
>
> I am seeing a deep symmetry between this work and the Resource Versioning
> work that I'll present on Wednesday. It turns out that we can represent a
> lot of the semantics needed for Resumeable Uploads through these generic
> Version and Parents headers that I'm proposing.
>
> Specifically, an upload can be represented as an append-only bytestream,
> which is growing over time until it reaches a committed "version." We can
> specify this with:
>
> Version-Type: bytestream
>
> This header defines that the resource grows at one byte per timestep, and
> thus "version =3D=3D length" in bytes.
>
> Now when a client needs to resume an upload, it simply asks the server
> what version it currently has:
>
> HEAD /upload
>
> And the server's response says what version it has:
>
> 200 OK
> Version: "500"
>
> This tells the client that the server is at time=3D500, thus it has the
> first 500 bytes of the resource, and the client now knows to upload a ran=
ge
> starting from 500:
>
> PUT /something
> Current-Version: "900"
> Parents: "500"
> Content-Range: bytes 500-900/900
> Content-Length: 400
>
> <binary data from 500-900>
>
> Check out the full example in Section 3.3 of
> draft-toomim-httpbis-versions-00
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions#sect=
ion-3.3>
> .
>
> The nice thing about this is that we can implement Resumeable Uploads
> without as many new headers. For instance, we wouldn't need this new
> Upload-Length
> <https://github.com/httpwg/http-extensions/issues/2832#issue-2416825177>
> header proposed today, because the Current-Version header in versions-00
> Section 2.6
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#s=
ection-2.6>
> already does the job =E2=80=94 you can see it on line 2 of the example PU=
T above!
>
> I think it would be great to have general headers that can be used for
> both Resource Versioning and Resumeable Uploads.
>
> What does the group think about this? Details are in versions-00, section
> 3.3
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#s=
ection-3.3>
> .
>
> Michael
> On 7/18/24 5:10 PM, Marius Kleidl wrote:
>
> Dear working group,
>
> last week we published iteration -04 for the resumable upload draft [1],
> which includes a number of useful additions:
>
>    - The application/partial-upload media type for PATCH requests
>    - Upload limits (max/min total size, max/min request size, upload
>    expiration)
>    - Guidelines for handling content and transfer codings
>    - Guidelines for integrity checks using digest fields
>    - Problem types for error responses
>
> From our perspective as editors, the draft is shaping up well, but a few
> small questions could still be addressed:
>
>
>    1. The server announces the upload limits (e.g. for maximum size or
>    minimum request size) only when the client creates a new upload resour=
ce.
>    For the client it would also be useful to know these limits upfront be=
fore
>    creating an upload. The client may then decide to not start an upload =
if it
>    cannot obey the limits. Should we allow the client to send an OPTIONS
>    request to retrieve the upload limits before creating a resource? [2]
>    2. If an upload is spread across multiple requests (e.g. due to limits
>    from intermediaries), the server only knows the upload length once the
>    upload is completed. However, in many cases the client knows the lengt=
h
>    upfront and could indicate the length to the server, who can use this
>    information to optimize the upload storage or reject the upload entire=
ly.
>    Should we add an advisory header to upload creation requests to indica=
te
>    the upload length? The Upload-Complete header would still keep the
>    authority for completing the upload. [3]
>    3. A client may include integrity fields to send the expected digest
>    for the upload to the server for later verification. This requires the
>    client to know the digest upfront or transmit it later using trailers
>    (which might not always be possible). Should we also provide a way for=
 the
>    client to request a digest from the server once the upload is complete=
?
>    Then the client and server can compute the digest during the upload an=
d the
>    client can compare its digest to the server's. [4]
>
> We are looking forward to discussing these points in the httpbis session
> on Monday!
>
> Best regards,
> Marius Kleidl
>
> [1]
> https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload=
-04
> [2] https://github.com/httpwg/http-extensions/issues/2833
> [3] https://github.com/httpwg/http-extensions/issues/2832
> [4] https://github.com/httpwg/http-extensions/issues/2834
>
>

--000000000000aad24e061de5a887
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Michael,</div><div><br></div><div>there are indeed=
 similarities in our approaches and I like the perspective with which you a=
re looking at the problem! At the end of the day, most approaches to resuma=
ble uploads use the same underlying concepts but framed with different term=
inology, field names or status codes.</div><div><br></div><div>Focusing spe=
cifically on the usage of your draft for uploads, I have a few comments:</d=
iv><div><br></div><div>1. What is the parent version of a resource that a c=
lient is uploading to? Requests have to set this header to ensure they are =
appending at the correct offset, but I am wondering what parent version wou=
ld be included in responses to HEAD requests. The example responses in sect=
ion 3.3 always use the 0 byte count.</div><div>2. Can a client upload from =
a streaming data source without knowing the entire file size upfront? Is th=
e including of the Current-Version header mandatory when starting an upload=
?</div><div>3. The example request for case B in section 3.3.2 includes the=
 Content-Range header, which appears redundant. Does its value carry additi=
onal information that is not covered by Parents, Current-Version or Content=
-Length?</div><div>4. Does the method allow the client to split an upload a=
cross multiple requests? In our experience, servers can have an upper limit=
 on the size of a single request, which requires a client to split an uploa=
d and send multiple consecutive requests to transfer the entire file.<br></=
div><div><br></div><div>Feel free to ignore them if you they fall outside o=
f the draft&#39;s intended scope. I am looking forward to your presentation=
 on Wednesday!</div><div><br></div><div>Best regards,</div><div>Marius Klei=
dl<br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"=
gmail_attr">On Tue, Jul 23, 2024 at 12:51=E2=80=AFAM Michael Toomim &lt;<a =
href=3D"mailto:toomim@gmail.com">toomim@gmail.com</a>&gt; wrote:<br></div><=
blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l=
eft:1px solid rgb(204,204,204);padding-left:1ex"><u></u>

 =20
   =20
 =20
  <div>
    <p>It was great seeing this progress on resumeable uploads in HTTPWG
      today!</p>
    <p>I am seeing a deep symmetry between this work and the Resource
      Versioning work that I&#39;ll present on Wednesday. It turns out that
      we can represent a lot of the semantics needed for Resumeable
      Uploads through these generic Version and Parents headers that I&#39;=
m
      proposing.</p>
    <p>Specifically, an upload can be represented as an append-only
      bytestream, which is growing over time until it reaches a
      committed &quot;version.&quot; We can specify this with:</p>
    <blockquote>
      <p>Version-Type: bytestream<br>
      </p>
    </blockquote>
    <p>This header defines that the resource grows at one byte per
      timestep, and thus &quot;version =3D=3D length&quot; in bytes.<br>
    </p>
    <p>Now when a client needs to resume an upload, it simply asks the
      server what version it currently has:</p>
    <blockquote>
      <p>HEAD /upload<br>
      </p>
    </blockquote>
    <p>And the server&#39;s response says what version it has:</p>
    <blockquote>
      <p>200 OK<br>
        Version: &quot;500&quot;</p>
    </blockquote>
    <p>This tells the client that the server is at time=3D500, thus it has
      the first 500 bytes of the resource, and the client now knows to
      upload a range starting from 500:</p>
    <blockquote>
      <p>PUT /something<br>
        Current-Version: &quot;900&quot;<br>
        Parents: &quot;500&quot;<br>
        Content-Range: bytes 500-900/900<br>
        Content-Length: 400<br>
        <br>
        &lt;binary data from 500-900&gt;</p>
    </blockquote>
    <p>Check out the full example in=C2=A0<a href=3D"https://datatracker.ie=
tf.org/doc/html/draft-toomim-httpbis-versions#section-3.3" target=3D"_blank=
">Section
        3.3 of draft-toomim-httpbis-versions-00</a>.</p>
    <p>The nice thing about this is that we can implement Resumeable
      Uploads without as many new headers. For instance, we wouldn&#39;t
      need this new <a href=3D"https://github.com/httpwg/http-extensions/is=
sues/2832#issue-2416825177" target=3D"_blank">Upload-Length</a>
      header proposed today, because the Current-Version header in <a href=
=3D"https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#=
section-2.6" target=3D"_blank">versions-00
        Section 2.6</a> already does the job =E2=80=94 you can see it on li=
ne 2
      of the example PUT above!<br>
    </p>
    <p>I think it would be great to have general headers that can be
      used for both Resource Versioning and Resumeable Uploads.</p>
    <p>What does the group think about this? Details are in <a href=3D"http=
s://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#section-=
3.3" target=3D"_blank">versions-00,
        section 3.3</a>.<br>
    </p>
    <p>Michael<br>
    </p>
    <div>On 7/18/24 5:10 PM, Marius Kleidl
      wrote:<br>
    </div>
    <blockquote type=3D"cite">
     =20
      <div dir=3D"ltr">
        <div>Dear working group,</div>
        <div><br>
        </div>
        <div>last week we published iteration -04 for the resumable
          upload draft [1], which includes a number of useful additions:</d=
iv>
        <div>
          <ul>
            <li>The application/partial-upload media type for PATCH
              requests</li>
            <li>Upload limits (max/min total size, max/min request size,
              upload expiration)</li>
            <li>Guidelines for handling content and transfer codings</li>
            <li>Guidelines for integrity checks using digest fields</li>
            <li>Problem types for error responses</li>
          </ul>
          <div>From our perspective as editors, the draft is shaping up
            well, but a few small questions could still be addressed:</div>
          <div><br>
          </div>
          <div>
            <ol>
              <li>The server announces the upload limits (e.g. for
                maximum size or minimum request size) only when the
                client creates a new upload resource. For the client it
                would also be useful to know these limits upfront before
                creating an upload. The client may then decide to not
                start an upload if it cannot obey the limits. Should we
                allow the client to send an OPTIONS request to retrieve
                the upload limits before creating a resource? [2]<br>
              </li>
              <li>If an upload is spread across multiple requests (e.g.
                due to limits from intermediaries), the server only
                knows the upload length once the upload is completed.
                However, in many cases the client knows the length
                upfront and could indicate the length to the server, who
                can use this information to optimize the upload storage
                or reject the upload entirely. Should we add an advisory
                header to upload creation requests to indicate the
                upload length? The Upload-Complete header would still
                keep the authority for completing the upload. [3]<br>
              </li>
              <li>A client may include integrity fields to send the
                expected digest for the upload to the server for later
                verification. This requires the client to know the
                digest upfront or transmit it later using trailers
                (which might not always be possible). Should we also
                provide a way for the client to request a digest from
                the server once the upload is complete? Then the client
                and server can compute the digest during the upload and
                the client can compare its digest to the server&#39;s. [4]<=
br>
              </li>
            </ol>
            <div>We are looking forward to discussing these points in
              the httpbis session on Monday!</div>
            <div><br>
            </div>
            <div>Best regards,</div>
            <div>Marius Kleidl</div>
            <div><br>
            </div>
            <div>[1] <a href=3D"https://datatracker.ietf.org/doc/html/draft=
-ietf-httpbis-resumable-upload-04" target=3D"_blank">https://datatracker.ie=
tf.org/doc/html/draft-ietf-httpbis-resumable-upload-04</a></div>
            <div>[2] <a href=3D"https://github.com/httpwg/http-extensions/i=
ssues/2833" target=3D"_blank">https://github.com/httpwg/http-extensions/iss=
ues/2833</a></div>
            <div>[3] <a href=3D"https://github.com/httpwg/http-extensions/i=
ssues/2832" target=3D"_blank">https://github.com/httpwg/http-extensions/iss=
ues/2832</a></div>
            <div>[4] <a href=3D"https://github.com/httpwg/http-extensions/i=
ssues/2834" target=3D"_blank">https://github.com/httpwg/http-extensions/iss=
ues/2834</a></div>
          </div>
        </div>
      </div>
    </blockquote>
  </div>

</blockquote></div>

--000000000000aad24e061de5a887--