Re: Prague side meeting: HTTP/2 concurrency and request cancellation (CVE-2023-44487)

[Willy Tarreau <[e-mail]>]

Hi Mark,

On Wed, Oct 11, 2023 at 10:45:13AM +1100, Mark Nottingham wrote:
> Martin has already drafted one proposal:
>   https://martinthomson.github.io/h2-stream-limits/draft-thomson-httpbis-h2-stream-limits.html

I find that there are interesting points here. Yesterday my coworker
Amaury who works on H3 explained to me how QUIC streams are flow-
controlled and I found the principle much safer. I thought we could
slightly emulate it by sending lots of SETTINGS frames and keeping
track of which one advertised what so that we know what the client
ACKs but that would be totally ugly and complicated.

I do like Martin's idea above however. The only thing is that it will
not protect servers against older clients, hence attackers would just
act like an older client. And if for safety we decide to start lower
with a small MAX_CONCURRENT_STREAMS then it would penalize modern
clients during the first round trip. Or maybe we'd advertise both a
low value for MAX_CONCURRENT_STREAMS for legacy clients immediately
followed by a MAX_STREAMS frame advertising a larger one ? If so we
still need to make sure that it will not cause massive RSTs from
clients between the two :-/ Or maybe something like this could work:

   MAX_CONCURRENT_STREAMS = 100
   MAX_STREAMS = 100
   MAX_CONCURRENT_STREAMS = 10

Older clients would learn 100 then 10, possibly dropping excess
streams, while new clients would learn MAX_STREAMS=100 and from
that point ignore MAX_CONCURRENT_STREAMS=10.

> Other discussions might touch on whether there are other measures (protocol
> mechanisms or otherwise) that clients and servers can take, how concurrency
> is exposed to calling code, and whether we can give better guidance about how
> concurrency violations should be handled.

A few of these were discussed already in the thread below opened by Cory
4 years ago, where it was discussed how to count streams vs limit, and
where I even mentioned this exact method of attack consisting in sending
HEADERS followed by RST_STREAM that would not change the total stream
count from the protocol perspective:

  https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar/0131.html

We already faced that issue long ago when multiple haproxy instances
were stacked on top of each other over H2 and too short timeouts on the
front would cause long series of HEADERS+RST_STREAM on the back before
the request had a chance to be processed due to the second layer being
configured with a nice value making it slower than the first one. What
we've been doing in haproxy against this is that instead of counting the
streams at the protocol level, we count attached ones at the application
layer: these are created at the same moment, but they're released once
they're aware of the close. And we stop processing new streams once the
configured limit is passed. This means that for a limit of 100 streams
for example, if we receive 100 HEADERS and their 100 respective
RST_STREAM, it will indeed create 100 streams that are immediately
orphaned (and closed from an H2 perspective), but the 101th HEADERS
frame will interrupt processing until some of these streams are
effectively closed and freed (and not just at the protocol layer).

And from what I've read below from Maxim Dounin, it seems like nginx
applies a very similar strategy (they use x2 margin instead of +1 but
the principle is the same, let streams finish first):

  https://mailman.nginx.org/pipermail/nginx-devel/2023-October/S36Q5HBXR7CAIMPLLPRSSSYR4PCMWILK.html

As such, I suspect that these approaches might be much more common than
what sensationalist mass media want us to believe and that the issue is
more a matter of implementation choices (i.e. resource management) than
of the protocol. We know that the protocol has some deficiencies that make
naive implementations easily attackable (this one, setting stream windows
of 1 byte during transfers, long series of 1-byte HEADERS+CONTINUATION,
various forms of HOL blocking such as zero-windows, unlimited number of
PUSH_PROMISE etc) but I think that once you care about your resource
usage you have to make some reasonable compromises on all of these.

With that said I'd love to plug that hole with an elegant mechanism
involving just an optional new frame ;-)

Cheers,
Willy