Re: Resumable Upload draft updates

[Michael Toomim <[e-mail]>]

It was great seeing this progress on resumeable uploads in HTTPWG today!

I am seeing a deep symmetry between this work and the Resource 
Versioning work that I'll present on Wednesday. It turns out that we can 
represent a lot of the semantics needed for Resumeable Uploads through 
these generic Version and Parents headers that I'm proposing.

Specifically, an upload can be represented as an append-only bytestream, 
which is growing over time until it reaches a committed "version." We 
can specify this with:

    Version-Type: bytestream

This header defines that the resource grows at one byte per timestep, 
and thus "version == length" in bytes.

Now when a client needs to resume an upload, it simply asks the server 
what version it currently has:

    HEAD /upload

And the server's response says what version it has:

    200 OK
    Version: "500"

This tells the client that the server is at time=500, thus it has the 
first 500 bytes of the resource, and the client now knows to upload a 
range starting from 500:

    PUT /something
    Current-Version: "900"
    Parents: "500"
    Content-Range: bytes 500-900/900
    Content-Length: 400

    <binary data from 500-900>

Check out the full example in Section 3.3 of 
draft-toomim-httpbis-versions-00 
<https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions#section-3.3>.

The nice thing about this is that we can implement Resumeable Uploads 
without as many new headers. For instance, we wouldn't need this new 
Upload-Length 
<https://github.com/httpwg/http-extensions/issues/2832#issue-2416825177> 
header proposed today, because the Current-Version header in versions-00 
Section 2.6 
<https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#section-2.6> 
already does the job — you can see it on line 2 of the example PUT above!

I think it would be great to have general headers that can be used for 
both Resource Versioning and Resumeable Uploads.

What does the group think about this? Details are in versions-00, 
section 3.3 
<https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-versions-00#section-3.3>.

Michael

On 7/18/24 5:10 PM, Marius Kleidl wrote:
> Dear working group,
>
> last week we published iteration -04 for the resumable upload draft 
> [1], which includes a number of useful additions:
>
>   * The application/partial-upload media type for PATCH requests
>   * Upload limits (max/min total size, max/min request size, upload
>     expiration)
>   * Guidelines for handling content and transfer codings
>   * Guidelines for integrity checks using digest fields
>   * Problem types for error responses
>
> From our perspective as editors, the draft is shaping up well, but a 
> few small questions could still be addressed:
>
>  1. The server announces the upload limits (e.g. for maximum size or
>     minimum request size) only when the client creates a new upload
>     resource. For the client it would also be useful to know these
>     limits upfront before creating an upload. The client may then
>     decide to not start an upload if it cannot obey the limits. Should
>     we allow the client to send an OPTIONS request to retrieve the
>     upload limits before creating a resource? [2]
>  2. If an upload is spread across multiple requests (e.g. due to
>     limits from intermediaries), the server only knows the upload
>     length once the upload is completed. However, in many cases the
>     client knows the length upfront and could indicate the length to
>     the server, who can use this information to optimize the upload
>     storage or reject the upload entirely. Should we add an advisory
>     header to upload creation requests to indicate the upload length?
>     The Upload-Complete header would still keep the authority for
>     completing the upload. [3]
>  3. A client may include integrity fields to send the expected digest
>     for the upload to the server for later verification. This requires
>     the client to know the digest upfront or transmit it later using
>     trailers (which might not always be possible). Should we also
>     provide a way for the client to request a digest from the server
>     once the upload is complete? Then the client and server can
>     compute the digest during the upload and the client can compare
>     its digest to the server's. [4]
>
> We are looking forward to discussing these points in the httpbis 
> session on Monday!
>
> Best regards,
> Marius Kleidl
>
> [1] 
> https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload-04
> [2] https://github.com/httpwg/http-extensions/issues/2833
> [3] https://github.com/httpwg/http-extensions/issues/2832
> [4] https://github.com/httpwg/http-extensions/issues/2834