Re: [Webtransport] Flow Control for high-throughput Streams

[David Schinazi <[e-mail]>]

(I'm still speaking as individual contributor)

I think your analysis ignores the fact that every deployed implementation
of h2 and h3 lies about flow control - none of them actually plan to
allocate the amount of memory they allow the peer to theoretically use in
flow control. I expect that WebTransport implementations will do the same -
ours will.

We have to balance theoretical correctness with complexity (both
implementation complexity, and standards development complexity). I'm
personally inclined to defer this problem to an extension until it becomes
an issue in practice. To add a data point: Google QUIC put all the headers
on the control stream (and used HPACK instead of QPACK), and while that's
theoretically bad, in practice it was fine. I'm strongly opposed to adding
complexity without demonstrated impact, and extensions are well suited to
that kind of experimentation.

But again, this was me speaking as an individual contributor.
As chair, I'm interested to hear what others think about this topic.

David

On Mon, Mar 25, 2024 at 12:03 PM Marten Seemann <[e-mail]>
wrote:

> > This issue can be solved by having the QUIC stream layer retransmit all
> of the unacked data on the WebTransport stream every time it tries to send
> a subsequent flow control frame.
>
> There are a number of things you can do at the QUIC layer. For example,
> you could also do some kind of FEC for the control stream (in the simplest
> case, you could retransmit every STREAM frame multiple times on a timer
> less than 1 RTT). My QUIC implementation doesn't have any API to enable
> this behavior, and I doubt many others do. In my stack, this would be far
> from trivial to implement.
>
> I'd argue that* if we need to change the (re)transmission logic of STREAM
> frames at the QUIC layer for this one use case, then using a QUIC stream is
> not the right abstraction* to begin with.
>
> > I'd like to see data showing that this is a problem in practice.
>
> To make things simple, let’s look at running HTTP on top of WebTransport
> streams. I’m not saying that this is most common use case of WebTransport,
> but it’s a protocol that we’re all familiar with, and that we actually
> already have data for.
>
> I’ll argue that the problem in WebTransport is roughly equivalent to the
> experiment Alan ran back in the 2017 (
> https://mailarchive.ietf.org/arch/msg/quic/S5p63SfjiKSaqObwUVO_fRGBOYA/).
> Back then we discussed if the HoL blocking introduced by running HPACK on a
> QUIC stream poses a problem.
>
> The scenarios are comparable when:
> * The client opens new streams to send new requests (same as in the
> original experiment).
> * At least some of the responses are not tiny, such that the session
> utilizes a large fraction of the current session flow control limit.
> * The client employs a flow control auto-tuning algorithm that aims to
> match the size of the flow control window of the session to ~ the BDP of
> the connection. As far as I know, this is what Chrome implements (
>
> https://docs.google.com/document/d/1F2YfdDXKpy20WVKJueEf4abn_LVZHhMUMS5gX6Pgjl4/edit
> ).
>
> The reason this scenario is roughly comparable to Alan’s experiment is:
> * Loss of a session-level flow control frame means that the server will
> run out of session-level flow control credit before the (1.x RTTs it takes
> until the) retransmission arrives.
> * Packet loss on the WebTransport control stream is the equivalent of the
> losing data on the HPACK stream. Packet loss doesn’t need to affect a
> MAX_DATA frame, any packet loss that blocks the server from consuming a
> MAX_DATA frame sent on the stream is sufficient.
> * Running out of session-level flow control means that no existing stream
> can make progress in downstream direction. And while the client can open
> new request streams, the server won’t be able to even send a single byte in
> response. This is equivalent to waiting for the HPACK message to be able to
> decode future requests.
>
> The effect on the processing of new requests will be similar to what was
> captured by the "Cumulative HOL Delay" metric.
>
> Back then, our conclusion of the discussion was that we couldn't live with
> the HoL blocking, and led to the design of QPACK:
> > Head-of-line blocking is a very real problem for serialized HPACK that
> gets predictably worse as RTT and loss increase. At a fairly modest 100ms
> RTT + 2% loss, HOL blocking on headers added an average of 40ms / request.
> At 200ms/5% loss, it’s almost 100ms/request, and the p100 (max) was
> 497ms/request.
>
>
> On Sun, 24 Mar 2024 at 17:16, David Schinazi <[e-mail]>
> wrote:
>
>> (I'm speaking as individual contributor)
>>
>> I'd like to see data showing that this is a problem in practice. If it
>> appears to be an issue, I'm not sure that changing the wire format is the
>> best solution. This issue can be solved by having the QUIC stream layer
>> retransmit all of the unacked data on the WebTransport stream every time it
>> tries to send a subsequent flow control frame. That would provide very
>> similar performance properties without complicating flow control for
>> everyone.
>>
>> David
>>
>> On Tue, Mar 19, 2024 at 7:47 PM Marten Seemann <[e-mail]>
>> wrote:
>>
>>> After the discussion we had in yesterday’s session, I’ve spent some more
>>> time thinking about how and in which situations the HoL blocking introduced
>>> by putting flow control capsules on a control stream might hurt
>>> performance. Unfortunately, this is not an edge case, but a fairly common
>>> scenario.
>>>
>>> Consider an application transferring a large amount of data on a single
>>> stream. In QUIC, the receiver would select or auto-tune the stream receive
>>> window to slightly exceed the connection's BDP, assuming it sends
>>> MAX_STREAM_DATA frames multiple times per RTT. This ensures that
>>> flow-control doesn’t block the transfer. If MAX_STREAM_DATA is sent less
>>> frequently, a window slightly larger than the BDP is needed to be able to
>>> fully utilize the available bandwidth.
>>>
>>> For connections not terminating at a CDN's edge, BDPs can be quite large
>>> due to the longer RTTs. For instance, a 200ms RTT connection with a
>>> bandwidth of 1 Gbps has a BDP of 25 MB.
>>>
>>> When packet loss occurs on the WebTransport control stream (in upstream
>>> direction), no capsules can be read until this packet loss is repaired,
>>> which takes at least (since loss detection might take some time as well) 1
>>> RTT. This applies even if the control stream is given the highest priority
>>> when sending. For a BDP-sized flow control window, this means that the
>>> high-throughput stream (in downstream direction) is now blocked on flow
>>> control until the retransmission arrives (i.e. for at least 1 RTT).
>>>
>>> The solution suggested in yesterday's session, "just use higher flow
>>> control limits," would mean increasing the stream flow control window to
>>> (at least!) 50 MB from 25 MB for our example. This approach seems
>>> unappealing from a resource allocation standpoint.
>>> --
>>> Webtransport mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/webtransport
>>>
>>