h2 proxy and connection flow control

[Tatsuhiro Tsujikawa <[e-mail]>]

I'm not sure this is already discussed, but I found a case where
stalling single stream will stall all streams in entire session
when a h2 proxy is involved while playing with h2 proxy.

Here is an example setup of the scenario:

+---------+            +----------+            +---------+
|         |+---------->| Frontend |+---------->|         |
| Browser |            | Proxy    |            | Backend |
|         |<----------+|          |<-----------|         |
+---------+            +----------+            +---------+

Browser, Frontend Proxy and Backend server all speak h2.  There
is 1 TCP connection between Browser and Frontend Proxy.  Also
there is 1 TCP connection between Frontend Proxy and Backend.
These connections are independent h2 sessions.

Browser uses 10MB receive window for stream and 10MB for
connection.  Frontend Proxy uses 64KB receive window for stream
and 64KB for connection.

Browser sends requests to Frontend Proxy, and then Frontend Proxy
issues request to Backend and sends back its response to Browser.

Now suppose that Browser issues a request to Frontend Proxy (this
is denoted by stream A).  Frontend Proxy relays its request to
Backend (this is denoted by stream B).  Suppose that response
body is 40MB.  Then Frontend Proxy sends back response for stream
A to Browser but at some point Browser stops sending stream
WIDOW_UPDATE for some reason (e.g. it is a streaming media and it
is not consumed by media player).  Eventually, Browser's receive
window for stream becomes 0.

Since Browser's receive window is 0, Frontend Proxy cannot send
more DATA for stream A to Browser.  Meanwhile, Frontend Proxy
continues to receive DATA for stream B from Backend.  Since
stream A's stream window is 0, Frontend Proxy cannot proceed and
just buffer the received data and eventually Frontend Proxy's
receive window for connection becomes 0.

After this happens, Browser can still issue new request, but
Backend cannot send response to Frontend Proxy because its recive
window is 0.  This means that if Browser stalls one stream, it
stalls all streams in entire session.

+---------+            +----------+            +---------+
|         |+---------->| Frontend |+---------->|         |
| Browser |            | Proxy    |            | Backend |
|         |<----------+|          |<-----------|         |
+---------+     ^      +----------+     ^      +---------+
                |                       |
             stream A                all streams
            was stalled              was stalled
            due to stream            due to connection
           window is 0               window is 0

If a connection between Frontend Proxy and Backend is shared by
multiple clients, then things get worse, because if one client
stalls one stream, it affects all clients sharing the connection.

I think stalling stream by Browser is reasonable behaviour and
not a bug, because it just means that data is not consumed
yet. Or else, this is just another "Don't do that" situation?

How do you deal with this issue? (or is it not an issue because
I'm completely wrong?)

Here is my observation:

o Effectively disable connection level flow control in proxy
  -- this is controversial because connection flow control is
     effective especially for proxy to limit its memory
     commitment.

o Actively timeout stalled stream by proxy
  -- this may work, but incurs delay before timeout.

o Innovate flow control algorithm not to cause this situation
  -- I'm not sure how it looks like.

Best regards,

Tatsuhiro Tsujikawa