Re: http/2 prioritization/fairness bug with proxies

[William Chan (陈智昌) <[e-mail]>]

There are a lot of emails here, and I haven't read them all. But this
discussion of browsers and multiple connections and bulk data transfers
worries me. Please remember what the browser vendors are telling you: we
want prioritization amongst HTTP requests involved in browsing, all of
which count as "interactive" and "non-bulk". Please see:
http://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar/0568.html
http://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar/0573.html

I don't understand how per-priority TCP connections will solve this.


On Wed, Feb 13, 2013 at 1:48 PM, Nico Williams <[e-mail]>wrote:

> On Wed, Feb 13, 2013 at 3:23 PM, Roberto Peon <[e-mail]> wrote:
> > On Wed, Feb 13, 2013 at 12:57 PM, Nico Williams <[e-mail]>
> > wrote:
> > Even if the network does the right thing and the bytes have arrived,
> TCP's
> > API still only lets you access the packets in-order.
>
> Are we brave enough to try SCTP instead of TCP for HTTP/2.0?
>
> I didn't think so.
>
> :) (that should be a sad smiley, actually)
>
> >> There's two ways to address these issues: either don't it (it ==
> >> multiplex diff QoS traffic over the same TCP conn.) or try hard never
> >> to write more than one BDP's worth of bulk without considering higher
> >> priority traffic.
> >
> > QoS for packets on multiple connections also doesn't work- each entity
> > owning a connection sends at what it believes is its max rate, induces
> > packet loss, gets throttled appropriately, and then takes too make RTTs
> to
> > recover. You end up not fully utilizing the channel(s).
>
> No, no, all bulk traffic should be sent over one connection at max
> rate.  Multiple bulk flows can be multiplexed safely over one TCP
> connection, therefore they should be.
>
> High priority traffic _generally_ means "non-bulk", therefore "max
> rate" for non-bulk is generally much, much less than for bulk and,
> therefore, non-bulk traffic can be multiplexed safely over a single
> TCP connection, being careful to move to a bulk connection when a
> non-bulk flow changes nature.
>
> The sender will know what whether a message is a bulk message or not.
>
> One complication here is that many requests will be non-bulk but their
> responses will be.  I.e., you might want to write the responses to
> requests on a different connection from the request!  And now you need
> an XID or some such, but you probably want one anyways so that
> responses can be interleaved.
>
> (For example by analogy, if we were talking about doing this as an
> SSHv2 extension we might migrate a pty stdout channel to a bulk
> connection when the user does a cat(1) of a huge file.  This is much
> harder in SSHv2 because we have logical octet streams for interactive,
> high-priority data, but we don't have such a thing in HTTP, so this is
> not a concern at all.  This is just an analogy to illustrate the
> point.)
>
> > The hard part is "considering higher priority traffic" when that traffic
> is
> > being send from a different machine, as would occur in the multiple
> > connection case.
>
> Are you talking about proxies aggregating traffic from multiple
> clients into one [set of] TCP connection[s] to a given server?  Sure,
> but all the proxy needs is to know whether a given request (or
> response) is bulk or not.
>
> > With a single connection, this is easy to coordinate. Agreed that
> estimating
> > BDP isn't trivial (however it is something that TCP effectively has to
> do).
>
> A single connection is a bad idea.  We already use multiple
> connections today in _browsers_.  Of course, for non-browser apps
> multiple connections may be quite a change, but that should be a)
> optional, b) acceptable anyways.
>
> >> > which we'd otherwise be able to do without. Unfortunately, per
> priority
> >> > TCP
> >> > connections don't work well for large loadbalancers where each of
> these
> >> > connections will likely be terminating at a different place. This
> would
> >> > create a difficult synchronization problem server side, full of races
> >> > and
> >> > complexity, and likely quite a bit worse in complexity than getting
> flow
> >> > control working well.
> >>
> >> I think you're saying that because of proxies it's difficult to ensure
> >> per-priority TCP connections, but this is HTTP/2.0 we're talking
> >> about.  We have the power to dictate that HTTP/2.0 proxies replicate
> >> the client's per-priority TCP connection scheme.
> >
> > No, I'm saying that it is somewhere between difficult and impossible to
> > ensure that separate connections from a client end up on one machine in
> the
> > modern loadbalancer world.
>
> I don't think it should be difficult, much less impossible, for
> HTTP/_2.0_.  What you need for this is to identify flows so their
> requests/responses can be grouped.  The main thing that comes to mind
> is that the load balancer needs to understand Chunked PUTs/POSTs and
> get them to go to the same end server -- surely this is handled
> already in HTTP/1.1 load balancers.
>
> > From a latency perspective, opening up the multiple connections can be a
> > loss as well-- it increases server load for both CPU and memory and
> vastly
> > increases the chance that you'll get a lost-packet on the SYN which takes
> > far longer to recover from as it requires an RTO before RTT has likely
> been
> > computed.
>
> Well, sure, but the sender could share one connection for multiple QoS
> traffic types while the additional connections come up, and hope for
> the best -- mostly it should work out.
>
> >> > Note that the recommendation will be that flow control be effectively
> >> > disabled unless you know what you're doing, and have a good reason
> >> > (memory
> >> > pressure) to use it.
> >>
> >> Huh?  Are you saying "we need and will specify flow control.  It won't
> >> work.  Therefore we'll have it off by default."  How can that help?!
> >> I don't see how it can.
> >
> > Everyone will be required to implement the flow control mechanism as a
> > sender.
> > Only those people who have effective memory limitations will require its
> use
> > when receiving (since the receiver dictates policy for flow control).
>
> So this is a source quench type flow control?  (As opposed to window
> size type, as in SSHv2.)  But note that the issue isn't the need to
> quench fast sources from slow sinks.  The issue is that by the time
> you notice that you have a source/sink bandwidth mismatch it's too
> late and TCP flow control has kicked in.  Of course, the receiver can
> recover by quenching the sender and then reading and buffering
> whatever's left on the wire, thus freeing up bandwidth on the wire for
> other sources, but the cost is lots of buffer space on the receiver,
> unless you can tell the sender to resend later and then you can throw
> away instead of buffer.
>
> Nico
> --
>
>