Re: Indeterminate-length partial content messages

[Austin William Wright <[e-mail]>]

> On Nov 2, 2023, at 22:48, Glenn Strauss <[e-mail]> wrote:
> 
> On Thu, Nov 02, 2023 at 09:26:17PM -0700, Austin William Wright wrote:
>> Hello HTTP WG,
>> 
>> HTTP APIs recently adopted my “Byte Range Patch” draft, a media type to overwrite or append to a resource, especially when Partial PUT is not suitable (e.g. when server support is undetermined, or when the patch must be exchanged as a file). It re-uses standard HTTP fields including Content-Range, but notably this field doesn’t support ranges of indeterminate length, so there's no way to encode an indefinitely long write at a specific offset—you must know the length of the write when you begin the request.
>> 
>> Currently, as a workaround, the draft specifies a special case for the Content-Range syntax. Since workarounds like this are questionable, I’ll specify a different field name (perhaps "Content-Offset"). However, this brings its own problems: There would be a field that only functions in the body of these PATCH requests, and isn’t used in HTTP headers or any other context. While it's plausible for Byte Range PATCH to abandon the HTTP field design, I think it may be simpler on the whole to share semantics with Range requests and multipart messages. And further, standardizing partial content offsets would have greater utility, including in 206 Partial Content responses, and synchronization more broadly.
>> 
>> This problem was previously observed in streaming media, treated by the experimental RFC 8673 <https://www.rfc-editor.org/rfc/rfc8673.html> (HTTP Random Access and Live Content). This RFC suggests using a very large Range endpoint to request the server stream content as it becomes available. Then, it uses this large number in the response Content-Range to indicate that the response is of indeterminate length.
>> 
>> While this solution caters to the requirements of Range requests, it cannot be used for uploads, where no Range field is used. And if a client sends “Content-Range: bytes 100-999999999999” in a request, but ends the stream with less content than that, this should only be seen as an error, not as an understanding that the exact length was unknown. So I think a different, more general solution is warranted.
>> 
>> This would also serve as an important building block for synchronization over HTTP, since this could be used in Partial PUT <https://www.rfc-editor.org/rfc/rfc9110.html#name-partial-put> or a Byte Range PATCH <https://datatracker.ietf.org/doc/draft-ietf-httpapi-patch-byterange/> to append to shift buffers <https://www.rfc-editor.org/rfc/rfc8673.html#name-shift-buffer-representation>, and other “live resources” whose content is not entirely known at request-time, but may be streamed as it becomes available (as opposed to transferring only the data defined at the time of the request). Features that could build on top of this may include:
>> 
>> - Indicating support for indeterminate-length 206 (Partial Content) responses.
>> - Indicating preference for “stream live data” versus “snapshot-at-request-time” messages.
>> - Managing sparse resources, including shift buffers and more complicated synchronization (e.g. multiple clients uploading to the same resource in parallel).
>> - Optimizing caching for shift buffers (e.g. indicate that content may grow and/or become forgotten, but does not change once defined).
>> - Subscribing to changes to an underlying resource (in realtime or as desired).
>> 
>> The two most obvious ways to define this feature would be (1) to extend Content-Range to a form like “bytes 10-*/*” (where the star indicates exact value unknown), or (2) a new header like “Content-Offset: 10” (or what Resumable Uploads calls “Upload-Offset” <https://httpwg.org/http-extensions/draft-ietf-httpbis-resumable-upload.html#name-upload-offset>).
>> 
>> I would like to propose "Content-Offset = sf-integer”, since there's a certain symmetry to it (other HTTP fields that change when the message is of indeterminate length). Though, in cachable responses, modifying Content-Range may be desirable instead, depending on how origins want to non-implementing caches to act. While some amount of compatibility must be considered (especially caches), I feel this is a problem that will spawn domain-specific solutions over and over until there’s a general solution.
>> 
>> Please send me feedback on this proposal, I would especially like to hear from anyone with experience implementing HTTP Random Access and Live Content, or with any of the use cases I’m describing here. Then if this seems reasonable, I’d can draft an experimental I-D.
> 
> When uploading a live stream, it is inefficient to send one byte of data
> at a time.  A quantum of data is sent instead.  This could be 1 second
> of audio/video, or could be 5 or 15 or 30 seconds.  Portable CD players
> back in the 1990's would read some 30 seconds of data off the disc so
> that skips could be recovered before the listener noticed.  But I
> digress.
> 
> For a server that is storing an upload, writes to disk occur in
> blocks, minimum 512 bytes (POSIX) and often 4k or larger.  Even if
> storing in memory, memory is allocated from the OS to the process
> in blocks of memory page sizes, often 4k or larger.
> 
> Sending a chunk of data, preferably block-sized or larger, is
> recommended for efficiency.

Yes, there’s all sorts of ways that the application may prefer to packetize the data, which are all supported, whether you use the standard response, RFC 8673 “live content", or Content-Offset.

> Appending to the end of a file is a special-case of patching a file.
> After all, O_APPEND is a POSIX flag to open(2).

This is a good point. Append is a special case in the sense that it’s a write offset at the length of the file.

However, I haven’t yet treated the case where you don’t know the length of the resource, but you want to append to it. For resumable uploads and other synchronization use, this isn’t a problem (and actually desired, you don’t want to leave any gaps). But for other uses like writing to a log file, the client may not care where the end is or if it gets mixed with other appends.

In this case, perhaps there should be an alternate form, like “Content-Offset: *” (where star represents current length of resource, though unknown). This could be mapped to O_APPEND.

> => Why must an "indeterminate length" be transmitted in a range request?
> Why not send a sequence of append requests, in 1-sec-of-data increments?

Two reasons:

First, we already have the ability to make indeterminate-length uploads and downloads from the beginning of a resource; why should we lose this ability because we want to skip over a portion of it? And while determinate-length messages may suffice for archived content, it is not practical for live streaming applications. If the client and the server are in the same room, the latency will cause humans some serious problems. If you’re trying to align video or speakers with each other, just 30ms is one television video frame, or 10m of sound distance (frequencies above 15Hz, the entire audible spectrum, will fall out-of-phase). At one second, conference calls become a circus.

Now the server could relay the data as soon as it comes in, before the end of the message arrives—but then why break up the stream into multiple requests in the first place? Just continue using the first one. HTTP is already widely used for live streaming in this manner, with PUT and POST, but there is no standard way to write at an offset, you can only start a stream from the beginning.

Second, clients cannot distinguish “live content” messages from “static content" in most cases. An origin server reading an image on a filesystem, a gateway forwarding from an origin, and a server relaying a live video feed, do not require different capabilities of the client. In all three cases, the data comes through at a certain maximum rate, and the final length might not be known by the server until the end.

Yet, in all these cases, a client might want to restart the request at a certain offset—but if the server doesn’t know the final length of the resource, then this query may be impossible to answer. (Strictly speaking: the server doesn’t have to know the total length of the resource, but it does have to know where the response will end before it can even begin.)

That is to say: Resuming or breaking up requests should always be possible, but never required. In the absence of an actual error, this should be at the discretion of the client.

> Separately, another problem with "indeterminate length" is file range
> locking, which might prevent scenarios of multiple uploads of different
> ranges.  Another is resource allocation on the server, indeterminant
> versus guaranteed allocation of specific size (think posix_fallocate()).

Yes, this should be a consideration. I don’t think a locking mechanism needs to be defined at this time, but it's one of the uses that should be afforded.

> When designing a synchronization mechanism, why is HTTP the right layer
> to implement this, versus using HTTP as an opaque transport layer,
> similar to TCP or UDP?

Implementing such a feature without HTTP would mean defining your own application protocol for synchronization. Not always unwarranted (especially if performance is the reason), but usually redundant.

Implementing this on top of HTTP would mean that generic clients (user agents and intermediaries alike) cannot use this feature. Most notably, resources couldn't be cached.

Cheers,

Austin.

> 
> Cheers, Glenn