Re: [Webtransport] Flow Control for high-throughput Streams

[Marten Seemann <[e-mail]>]

> This issue can be solved by having the QUIC stream layer retransmit all
of the unacked data on the WebTransport stream every time it tries to send
a subsequent flow control frame.

There are a number of things you can do at the QUIC layer. For example, you
could also do some kind of FEC for the control stream (in the simplest
case, you could retransmit every STREAM frame multiple times on a timer
less than 1 RTT). My QUIC implementation doesn't have any API to enable
this behavior, and I doubt many others do. In my stack, this would be far
from trivial to implement.

I'd argue that* if we need to change the (re)transmission logic of STREAM
frames at the QUIC layer for this one use case, then using a QUIC stream is
not the right abstraction* to begin with.

> I'd like to see data showing that this is a problem in practice.

To make things simple, let’s look at running HTTP on top of WebTransport
streams. I’m not saying that this is most common use case of WebTransport,
but it’s a protocol that we’re all familiar with, and that we actually
already have data for.

I’ll argue that the problem in WebTransport is roughly equivalent to the
experiment Alan ran back in the 2017 (
https://mailarchive.ietf.org/arch/msg/quic/S5p63SfjiKSaqObwUVO_fRGBOYA/).
Back then we discussed if the HoL blocking introduced by running HPACK on a
QUIC stream poses a problem.

The scenarios are comparable when:
* The client opens new streams to send new requests (same as in the
original experiment).
* At least some of the responses are not tiny, such that the session
utilizes a large fraction of the current session flow control limit.
* The client employs a flow control auto-tuning algorithm that aims to
match the size of the flow control window of the session to ~ the BDP of
the connection. As far as I know, this is what Chrome implements (
https://docs.google.com/document/d/1F2YfdDXKpy20WVKJueEf4abn_LVZHhMUMS5gX6Pgjl4/edit
).

The reason this scenario is roughly comparable to Alan’s experiment is:
* Loss of a session-level flow control frame means that the server will run
out of session-level flow control credit before the (1.x RTTs it takes
until the) retransmission arrives.
* Packet loss on the WebTransport control stream is the equivalent of the
losing data on the HPACK stream. Packet loss doesn’t need to affect a
MAX_DATA frame, any packet loss that blocks the server from consuming a
MAX_DATA frame sent on the stream is sufficient.
* Running out of session-level flow control means that no existing stream
can make progress in downstream direction. And while the client can open
new request streams, the server won’t be able to even send a single byte in
response. This is equivalent to waiting for the HPACK message to be able to
decode future requests.

The effect on the processing of new requests will be similar to what was
captured by the "Cumulative HOL Delay" metric.

Back then, our conclusion of the discussion was that we couldn't live with
the HoL blocking, and led to the design of QPACK:
> Head-of-line blocking is a very real problem for serialized HPACK that
gets predictably worse as RTT and loss increase. At a fairly modest 100ms
RTT + 2% loss, HOL blocking on headers added an average of 40ms / request.
At 200ms/5% loss, it’s almost 100ms/request, and the p100 (max) was
497ms/request.


On Sun, 24 Mar 2024 at 17:16, David Schinazi <[e-mail]>
wrote:

> (I'm speaking as individual contributor)
>
> I'd like to see data showing that this is a problem in practice. If it
> appears to be an issue, I'm not sure that changing the wire format is the
> best solution. This issue can be solved by having the QUIC stream layer
> retransmit all of the unacked data on the WebTransport stream every time it
> tries to send a subsequent flow control frame. That would provide very
> similar performance properties without complicating flow control for
> everyone.
>
> David
>
> On Tue, Mar 19, 2024 at 7:47 PM Marten Seemann <[e-mail]>
> wrote:
>
>> After the discussion we had in yesterday’s session, I’ve spent some more
>> time thinking about how and in which situations the HoL blocking introduced
>> by putting flow control capsules on a control stream might hurt
>> performance. Unfortunately, this is not an edge case, but a fairly common
>> scenario.
>>
>> Consider an application transferring a large amount of data on a single
>> stream. In QUIC, the receiver would select or auto-tune the stream receive
>> window to slightly exceed the connection's BDP, assuming it sends
>> MAX_STREAM_DATA frames multiple times per RTT. This ensures that
>> flow-control doesn’t block the transfer. If MAX_STREAM_DATA is sent less
>> frequently, a window slightly larger than the BDP is needed to be able to
>> fully utilize the available bandwidth.
>>
>> For connections not terminating at a CDN's edge, BDPs can be quite large
>> due to the longer RTTs. For instance, a 200ms RTT connection with a
>> bandwidth of 1 Gbps has a BDP of 25 MB.
>>
>> When packet loss occurs on the WebTransport control stream (in upstream
>> direction), no capsules can be read until this packet loss is repaired,
>> which takes at least (since loss detection might take some time as well) 1
>> RTT. This applies even if the control stream is given the highest priority
>> when sending. For a BDP-sized flow control window, this means that the
>> high-throughput stream (in downstream direction) is now blocked on flow
>> control until the retransmission arrives (i.e. for at least 1 RTT).
>>
>> The solution suggested in yesterday's session, "just use higher flow
>> control limits," would mean increasing the stream flow control window to
>> (at least!) 50 MB from 25 MB for our example. This approach seems
>> unappealing from a resource allocation standpoint.
>> --
>> Webtransport mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/webtransport
>>
>