Re: http/2 prioritization/fairness bug with proxies

[Roberto Peon <[e-mail]>]

Gotcha, yes, though hopefully that wouldn't' happen if any
re-prioritization messages were passed back to the source of the packets!

I think that intermediaries need outgoing buffers to be BDP for the
outbound size, and incoming buffers to be BDP for the incoming path :)
That *should* sum up to the BDP of the complete path, but arguably could be
different if the proxy induces processing delay (potentially
intentionally...)
-=R


On Mon, Feb 11, 2013 at 3:35 PM, Martin Thomson <[e-mail]>wrote:

> My bad, I jumped from that to the next point.  That is, once you start
> de-prioritizing streams, you have to have a mechanism to prevent you
> from being overwhelmed by incoming packets on that stream.
>
> Note also that the proxy can apply prioritization based on more than
> just the priority field.  It can actively avoid starvation based on
> more than just the priority field.
>
> (On a random hunch, could intermediaries need flow control windows
> sized for the BDP of the complete path from origin to client?)
>
> On 11 February 2013 15:27, Roberto Peon <[e-mail]> wrote:
> > If that is what you're saying, then I definitely misunderstood. I thought
> > you were advocating using flow control to reduce the effective priority
> of a
> > long-running item?
> > -=R
> >
> >
> > On Mon, Feb 11, 2013 at 3:21 PM, Martin Thomson <
> [e-mail]>
> > wrote:
> >>
> >> That's not at all what I had in mind, just a general re-iteration of
> >> the basic principles:
> >>
> >> Primarily:  Send bits based on availability and relative priorities.
> >> Secondarily:  Advertise flow control window availability based on when
> >> those bits are consumed (i.e. sent onwards).
> >>
> >> On 11 February 2013 14:57, Roberto Peon <[e-mail]> wrote:
> >> > I think that the overall complexity spend is lower when we put the
> >> > complexity into prioritization mechanisms instead of stuffing
> heuristics
> >> > into the flow control... (because that is just ewww and ouch in so
> many
> >> > ways..)
> >> >
> >> > -=R
> >> >
> >> >
> >> > On Mon, Feb 11, 2013 at 2:53 PM, Martin Thomson
> >> > <[e-mail]>
> >> > wrote:
> >> >>
> >> >> I'm thinking a simple mechanism.  The proxy can feed requests back to
> >> >> clients based on its own prioritization and use flow control windows
> >> >> on the back-end connections to ensure that it doesn't have to buffer
> >> >> infinitely.  Basically, the same sorts of logic that would be needed
> >> >> by a proxy that serves multiple clients.
> >> >>
> >> >> Yes, this is sub-optimal because it leads to either poor bandwidth
> >> >> utilization, lots of buffering, or worse.
> >> >>
> >> >> Sure, it's very easy to get this wrong.  I don't believe it
> >> >> unreasonable to consider this sort of thing to be marked "hard hat
> >> >> area".  In both directions.
> >> >>
> >> >> On 11 February 2013 14:46, William Chan (陈智昌) <[e-mail]
> >
> >> >> wrote:
> >> >> > Theoretically possible is one thing. But the moment we get into the
> >> >> > game
> >> >> > of
> >> >> > trying to carve up portions of BDP via per-stream flow control
> >> >> > windows
> >> >> > for
> >> >> > prioritization purposes in normal operation (as opposed to just
> >> >> > trying
> >> >> > to
> >> >> > make reasonable progress under excessive load), I think we're in
> >> >> > trouble,
> >> >> > and likely to get into performance issues due to poor
> >> >> > implementations.
> >> >> > As
> >> >> > I've stated before, I hope most implementations (and believe we
> >> >> > should
> >> >> > add
> >> >> > recommendations for this behavior) only use flow control (if they
> use
> >> >> > it
> >> >> > at
> >> >> > all, which hopefully they don't because it's hard) for maintaining
> >> >> > reasonable buffer sizes.
> >> >> >
> >> >> >
> >> >> > On Mon, Feb 11, 2013 at 2:39 PM, Martin Thomson
> >> >> > <[e-mail]>
> >> >> > wrote:
> >> >> >>
> >> >> >> Keep in mind that it is always possible for the intermediary to
> >> >> >> apply
> >> >> >> flow control to the infinite length stream so that, regardless of
> >> >> >> priority, it doesn't consume more than its fair share.
> >> >> >>
> >> >> >> On 10 February 2013 14:40, William Chan (陈智昌)
> >> >> >> <[e-mail]>
> >> >> >> wrote:
> >> >> >> > First, I totally agree SPDY/4 prioritization changes are far
> more
> >> >> >> > reaching.
> >> >> >> > Let's not talk about them yet. I share your complexity concern
> and
> >> >> >> > agree
> >> >> >> > we
> >> >> >> > need data before proceeding with many of those features, and I
> >> >> >> > plan
> >> >> >> > on
> >> >> >> > getting data.
> >> >> >> >
> >> >> >> > As far as grouping, I think it's definitely a bug for the case
> >> >> >> > where
> >> >> >> > you
> >> >> >> > have a forward proxy with users sharing the same HTTP/2.0
> >> >> >> > connection
> >> >> >> > to
> >> >> >> > an
> >> >> >> > origin server. You can have many high priority short-lived
> streams
> >> >> >> > which
> >> >> >> > can
> >> >> >> > starve out others, unless we implement the vague notion of
> "don't
> >> >> >> > starve
> >> >> >> > out
> >> >> >> > streams", which is difficult when it might be a sustained rate
> of
> >> >> >> > high
> >> >> >> > priority short-lived streams. It's easier in your latter case of
> >> >> >> > infinite,
> >> >> >> > high-priority streams.
> >> >> >> >
> >> >> >> > You're right that the high priority, infinite stream can starve
> >> >> >> > streams
> >> >> >> > within the same group. I don't think this means that grouping is
> >> >> >> > not
> >> >> >> > required, but that grouping is potentially insufficient. For
> >> >> >> > intragroup
> >> >> >> > starvation, I think it's debatable about whether the server
> should
> >> >> >> > be
> >> >> >> > smart
> >> >> >> > and not allow streams to _completely_ starve other streams
> within
> >> >> >> > a
> >> >> >> > group,
> >> >> >> > or that clients should have a reprioritization facility. I think
> >> >> >> > this
> >> >> >> > is
> >> >> >> > a
> >> >> >> > discussion worth having, but I'd personally classify it as
> >> >> >> > separate
> >> >> >> > from
> >> >> >> > whether or not the grouping feature is necessary.
> >> >> >> >
> >> >> >> > On Sun, Feb 10, 2013 at 2:06 PM, Mike Belshe <[e-mail]>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Mon, Feb 4, 2013 at 5:47 PM, William Chan (陈智昌)
> >> >> >> >> <[e-mail]>
> >> >> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> I'm sorry if I am unclear in any way. Please continue to
> >> >> >> >>> challenge/question my comments/assertions so I can clarify my
> >> >> >> >>> position
> >> >> >> >>> as appropriate.
> >> >> >> >>>
> >> >> >> >>> Just to be clear here, I stand by that it's a protocol bug
> >> >> >> >>> currently.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> +0.5.  :-)
> >> >> >> >>
> >> >> >> >> Mark - I think we could open a issue ticket with the current
> >> >> >> >> HTTP/2.0
> >> >> >> >> draft that this is a bug which will present itself for servers
> >> >> >> >> that
> >> >> >> >> implement naively.  This isn't strictly a "bug", since the spec
> >> >> >> >> says
> >> >> >> >> the
> >> >> >> >> server can do whatever it wants with priorities, but this is
> >> >> >> >> subtle
> >> >> >> >> enough
> >> >> >> >> (surfacing primarily in http/2 -> http/2 proxy situations),
> that
> >> >> >> >> many
> >> >> >> >> server
> >> >> >> >> implementors won't think of it unless we mention it in the
> spec.
> >> >> >> >>
> >> >> >> >> The bare minimum would be to simply document it and tell
> servers
> >> >> >> >> not
> >> >> >> >> to
> >> >> >> >> starve out streams.  However, this is probably a wimpy approach
> >> >> >> >> and
> >> >> >> >> I
> >> >> >> >> think
> >> >> >> >> we can do better.
> >> >> >> >>
> >> >> >> >> The SPDY/4 prioritization changes are far more reaching than
> just
> >> >> >> >> fixing
> >> >> >> >> this bug.  While I like grouping as a feature, I don't believe
> it
> >> >> >> >> actually
> >> >> >> >> fixes this bug:  a browser could open a high priority, infinite
> >> >> >> >> stream,
> >> >> >> >> which is competing across a shared proxy backend with other
> >> >> >> >> streams;
> >> >> >> >> unless
> >> >> >> >> the user manually switches tabs (or does something to force
> >> >> >> >> changed
> >> >> >> >> groups),
> >> >> >> >> the starvation can still occur.
> >> >> >> >>
> >> >> >> >> Mike
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>>
> >> >> >> >>> I agree with adding more hooks to convey advisory priority
> >> >> >> >>> semantics.
> >> >> >> >>> That said, "advisory" is open to interpretation. I agree that
> >> >> >> >>> the
> >> >> >> >>> sender should ultimately be in control of how it orders
> >> >> >> >>> responses,
> >> >> >> >>> and
> >> >> >> >>> indeed there are of course many situations where it's best for
> >> >> >> >>> the
> >> >> >> >>> sender to ignore the advisory priority. Yet, if the advisory
> >> >> >> >>> priority
> >> >> >> >>> semantics are generally not respected, then clients will not
> be
> >> >> >> >>> able
> >> >> >> >>> to rely on them, and will be forced to implement
> prioritization
> >> >> >> >>> at
> >> >> >> >>> a
> >> >> >> >>> higher layer, which suffers from the link underutilization vs
> >> >> >> >>> contention tradeoff I highlighted earlier.
> >> >> >> >>>
> >> >> >> >>> I appreciate the concern that we're adding complexity by
> >> >> >> >>> introducing
> >> >> >> >>> new semantics. I am arguing that because the existing
> mechanisms
> >> >> >> >>> for
> >> >> >> >>> addressing starvation are suboptimal, we should treat this as
> a
> >> >> >> >>> protocol bug and thus change the protocol in such a way as to
> >> >> >> >>> fix
> >> >> >> >>> this
> >> >> >> >>> problem. My suggestion for doing so was adding new priority
> >> >> >> >>> "grouping"
> >> >> >> >>> semantics. I am hopeful that these new semantics will not
> >> >> >> >>> introduce
> >> >> >> >>> an
> >> >> >> >>> inordinate amount of specification, as the primary idea is
> that
> >> >> >> >>> the
> >> >> >> >>> current SPDY priority levels would apply within a "group". I
> >> >> >> >>> think
> >> >> >> >>> we
> >> >> >> >>> can come up with a way to define a group that will be
> relatively
> >> >> >> >>> easy
> >> >> >> >>> to spec.
> >> >> >> >>>
> >> >> >> >>> SPDY/4 introduces other prioritization semantics beyond just
> >> >> >> >>> grouping,
> >> >> >> >>> but I wanted to focus on this one first, as I believe this is
> a
> >> >> >> >>> bug
> >> >> >> >>> that we *need* to fix. The other SPDY/4 priority changes are
> of
> >> >> >> >>> a
> >> >> >> >>> performance optimization nature, and I believe they will need
> to
> >> >> >> >>> be
> >> >> >> >>> justified by data. I have no plans to raise them up in this
> >> >> >> >>> group
> >> >> >> >>> until we have said data.
> >> >> >> >>>
> >> >> >> >>> On Tue, Feb 5, 2013 at 5:34 AM, Poul-Henning Kamp
> >> >> >> >>> <[e-mail]>
> >> >> >> >>> wrote:
> >> >> >> >>> > Content-Type: text/plain; charset=ISO-8859-1
> >> >> >> >>> > --------
> >> >> >> >>> > In message
> >> >> >> >>> > <[e-mail]>,
> >> >> >> >>> > Ben Nive
> >> >> >> >>> > n-Jenkins writes:
> >> >> >> >>> >
> >> >> >> >>> >>So the idea is the protocol contains enough 'hooks' to
> >> >> >> >>> >> sufficiently
> >> >> >> >>> >>express the different priorities between & within groups
> that
> >> >> >> >>> >> folks
> >> >> >> >>> >>would like to express but isn't prescriptive about how
> anyone
> >> >> >> >>> >> uses
> >> >> >> >>> >> or
> >> >> >> >>> >>implements different prioritisation, scheduling, etc
> schemes.
> >> >> >> >>> >
> >> >> >> >>> > That was clearly not how the original poster presented it:
> >> >> >> >>> >
> >> >> >> >>> >         "I consider all those options as suboptimal, and
> thus
> >> >> >> >>> >         consider this issue to be a protocol bug. Our SPDY/4
> >> >> >> >>> >         prioritization proposal addresses this by [...]"
> >> >> >> >>> >
> >> >> >> >>> > --
> >> >> >> >>> > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> >> >> >> >>> > [e-mail]         | TCP/IP since RFC 956
> >> >> >> >>> > FreeBSD committer       | BSD since 4.3-tahoe
> >> >> >> >>> > Never attribute to malice what can adequately be explained
> by
> >> >> >> >>> > incompetence.
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >
> >> >> >
> >> >> >
> >> >>
> >> >
> >
> >
>