Re: Resumable Upload draft updates

[Marius Kleidl <[e-mail]>]

Hi Michael,

there are indeed similarities in our approaches and I like the perspective
with which you are looking at the problem! At the end of the day, most
approaches to resumable uploads use the same underlying concepts but framed
with different terminology, field names or status codes.

Focusing specifically on the usage of your draft for uploads, I have a few
comments:

1. What is the parent version of a resource that a client is uploading to?
Requests have to set this header to ensure they are appending at the
correct offset, but I am wondering what parent version would be included in
responses to HEAD requests. The example responses in section 3.3 always use
the 0 byte count.
2. Can a client upload from a streaming data source without knowing the
entire file size upfront? Is the including of the Current-Version header
mandatory when starting an upload?
3. The example request for case B in section 3.3.2 includes the
Content-Range header, which appears redundant. Does its value carry
additional information that is not covered by Parents, Current-Version or
Content-Length?
4. Does the method allow the client to split an upload across multiple
requests? In our experience, servers can have an upper limit on the size of
a single request, which requires a client to split an upload and send
multiple consecutive requests to transfer the entire file.

Feel free to ignore them if you they fall outside of the draft's intended
scope. I am looking forward to your presentation on Wednesday!

Best regards,
Marius Kleidl

On Tue, Jul 23, 2024 at 12:51 AM Michael Toomim <[e-mail]> wrote:

> It was great seeing this progress on resumeable uploads in HTTPWG today!
>
> I am seeing a deep symmetry between this work and the Resource Versioning
> work that I'll present on Wednesday. It turns out that we can represent a
> lot of the semantics needed for Resumeable Uploads through these generic
> Version and Parents headers that I'm proposing.
>
> Specifically, an upload can be represented as an append-only bytestream,
> which is growing over time until it reaches a committed "version." We can
> specify this with:
>
> Version-Type: bytestream
>
> This header defines that the resource grows at one byte per timestep, and
> thus "version == length" in bytes.
>
> Now when a client needs to resume an upload, it simply asks the server
> what version it currently has:
>
> HEAD /upload
>
> And the server's response says what version it has:
>
> 200 OK
> Version: "500"
>
> This tells the client that the server is at time=500, thus it has the
> first 500 bytes of the resource, and the client now knows to upload a range
> starting from 500:
>
> PUT /something
> Current-Version: "900"
> Parents: "500"
> Content-Range: bytes 500-900/900
> Content-Length: 400
>
> <binary data from 500-900>
>
> Check out the full example in Section 3.3 of
> draft-toomim-httpbis-versions-00
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions#section-3.3>
> .
>
> The nice thing about this is that we can implement Resumeable Uploads
> without as many new headers. For instance, we wouldn't need this new
> Upload-Length
> <https://github.com/httpwg/http-extensions/issues/2832#issue-2416825177>
> header proposed today, because the Current-Version header in versions-00
> Section 2.6
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#section-2.6>
> already does the job — you can see it on line 2 of the example PUT above!
>
> I think it would be great to have general headers that can be used for
> both Resource Versioning and Resumeable Uploads.
>
> What does the group think about this? Details are in versions-00, section
> 3.3
> <https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#section-3.3>
> .
>
> Michael
> On 7/18/24 5:10 PM, Marius Kleidl wrote:
>
> Dear working group,
>
> last week we published iteration -04 for the resumable upload draft [1],
> which includes a number of useful additions:
>
>    - The application/partial-upload media type for PATCH requests
>    - Upload limits (max/min total size, max/min request size, upload
>    expiration)
>    - Guidelines for handling content and transfer codings
>    - Guidelines for integrity checks using digest fields
>    - Problem types for error responses
>
> From our perspective as editors, the draft is shaping up well, but a few
> small questions could still be addressed:
>
>
>    1. The server announces the upload limits (e.g. for maximum size or
>    minimum request size) only when the client creates a new upload resource.
>    For the client it would also be useful to know these limits upfront before
>    creating an upload. The client may then decide to not start an upload if it
>    cannot obey the limits. Should we allow the client to send an OPTIONS
>    request to retrieve the upload limits before creating a resource? [2]
>    2. If an upload is spread across multiple requests (e.g. due to limits
>    from intermediaries), the server only knows the upload length once the
>    upload is completed. However, in many cases the client knows the length
>    upfront and could indicate the length to the server, who can use this
>    information to optimize the upload storage or reject the upload entirely.
>    Should we add an advisory header to upload creation requests to indicate
>    the upload length? The Upload-Complete header would still keep the
>    authority for completing the upload. [3]
>    3. A client may include integrity fields to send the expected digest
>    for the upload to the server for later verification. This requires the
>    client to know the digest upfront or transmit it later using trailers
>    (which might not always be possible). Should we also provide a way for the
>    client to request a digest from the server once the upload is complete?
>    Then the client and server can compute the digest during the upload and the
>    client can compare its digest to the server's. [4]
>
> We are looking forward to discussing these points in the httpbis session
> on Monday!
>
> Best regards,
> Marius Kleidl
>
> [1]
> https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload-04
> [2] https://github.com/httpwg/http-extensions/issues/2833
> [3] https://github.com/httpwg/http-extensions/issues/2832
> [4] https://github.com/httpwg/http-extensions/issues/2834
>
>