Re: http/2 prioritization/fairness bug with proxies

Amos Jeffries <> Fri, 15 February 2013 06:43 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id A165621F87FE for <>; Thu, 14 Feb 2013 22:43:00 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -10.3
X-Spam-Status: No, score=-10.3 tagged_above=-999 required=5 tests=[AWL=0.299, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id DzSHCBjJpHOn for <>; Thu, 14 Feb 2013 22:42:59 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id C934121F8777 for <>; Thu, 14 Feb 2013 22:42:59 -0800 (PST)
Received: from lists by with local (Exim 4.72) (envelope-from <>) id 1U6Ez8-0004gA-Af for; Fri, 15 Feb 2013 06:41:26 +0000
Resent-Date: Fri, 15 Feb 2013 06:41:26 +0000
Resent-Message-Id: <>
Received: from ([]) by with esmtp (Exim 4.72) (envelope-from <>) id 1U6Eyv-0004et-Fr for; Fri, 15 Feb 2013 06:41:13 +0000
Received: from ([] by with esmtp (Exim 4.72) (envelope-from <>) id 1U6Eyu-0001Bl-8O for; Fri, 15 Feb 2013 06:41:13 +0000
Received: from [] (unknown []) by (Postfix) with ESMTP id 85F98E6EE3 for <>; Fri, 15 Feb 2013 19:40:48 +1300 (NZDT)
Message-ID: <>
Date: Fri, 15 Feb 2013 19:40:44 +1300
From: Amos Jeffries <>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2
MIME-Version: 1.0
References: <> <> <> <> <> <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=;;
X-W3C-Hub-Spam-Status: No, score=-3.4
X-W3C-Hub-Spam-Report: AWL=-3.353, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001
X-W3C-Scan-Sig: 1U6Eyu-0001Bl-8O 9f30fa6567e9f63303f339e634276f71
Subject: Re: http/2 prioritization/fairness bug with proxies
Archived-At: <>
X-Mailing-List: <> archive/latest/16609
Precedence: list
List-Id: <>
List-Help: <>
List-Post: <>
List-Unsubscribe: <>

On 15/02/2013 9:26 a.m., Nico Williams wrote:
> On Wed, Feb 13, 2013 at 4:43 PM, Roberto Peon <> wrote:
>> SCTP: Unfortunately not deployable due to consumer NAT interactions.
> I know :(

That being todays problem.  Are these NAT interactions expected to still 
be widespread into the 10-20 years future lifetime of HTTP/2. Noting 
that these NAT obstacles do not exist in some networks already, and with 
IPv6 rollout are declining in popularity over time.

ie, is it worth the effort to split the Draft by transport under-layer 
and document HTTP/2 over SCTP as well as over TCP?

>> Bulk-traffic: There are a number of different levels of traffic we're
>> prioritizing. It isn't just 'bulk' or 'highpri'
> I believe there's really only two or three categories of traffic:
> bulk, non-bulk w/ Nagle algorithm, non-bulk w/o Nagle.  That's really
> it.

I was party to some HTTP object size evaluations a few years back. There 
were at the time four 'bumps' in the average object size. Roughly:

0-64 Byte entity - common to all networks, this roughly matches:
   * the HTTP/1.1 prevalence of 0-sized 304 validation messages,
   * the 1990's habitual breaking of images into thousands of <128 byte 
   * the 2006+ habit of using compressed PNG icons for graphic design.

4-40 KB entity (network dependent peak around 8K, 11K, or a flatish 15K-31K)
   * HTML and images on websites optimized for speed or client experience
   * CSS and early 2000's JavaScript files.

10-20 MB entity
  * MP3 and WAV audio files from that early 2000's fad on embeded sound
  * video files from YouTube and similar video sites.
  * short streams and podcasts.
  * some backup systems tarball or equivalent file transfers

0.5 - 8 GB entity
  * CD and DVD images
  * VM image transfers
  * bulk archive transfers between servers
  * streaming media

These varied between the few networks I got a good look at, and there 
were of course a lot of traffic in the gaps between but the overall 
graph looks like a declining cosine wave.

I think there are at least 3 priority types identified in the above 
sets, and I'm not sure where all the long-polling 'services' and such 
will fit in once they are identified. Perhapse they were already 
contributing but not noticed in the flood of web-page traffic?

>    If there are multiple bulk flows where it is not desirable for
> one slow/stuck sink to cause all the other bulk flows to stop, then
> you need a TCP connection per-bulk flow (or at least that one
> possibly-slow flow).
> But it's possible that we're talking about different thing.  One thing
> is priority for server processing of requests.  Another is for proxies
> -- here we have to start worrying about the multiplexing issues that
> SSHv1 and SSHv2 have had, and since I think we ware talking about
> proxies I keep coming back to these issues.
>> Certain features require synchronization between control data and payload
>> (e.g. server push).
>> It is not possible to demux these without additional complexity from a
>> protoco standpoint.
> I don't see why.  Can you explain in more detail?

Given two connections between client A and "" (one for high 
priority control frames, the other for large bulk frames). If client A 
wants to abort a push it emits a control frame to stop it. Sounds 
simple, BUT...

* What if the control frame priority channel was different to the 
priority of the channel it is trying to stop? - what if those are 
different endpoints without the client A knowing? Note that when there 
is more than just client<->origin involved the client _can't_ know TCP 
level details about what the middle is doing.

* Do we let critical control frames get HOL blocked (or frame-size 
blocked at least) on the bulk channel just because they refer to that 
channels content? - if so what is the use of having priority in the 
first place and not just splitting requests over multiple connections?

* if the control frame does reach a single origin, but via two 
middleware paths... what are the middleware which never reach the STOP 
frame expected to do? what if middleware on the bulk path wants to emit 
STOP by itself? now the origin must handle control frames from multipe 
channels from independent sources.

... complexity. And not just complexity. That last case with middleware 
policing transfer size adds security aspects as well.

Lets not get buried in this ants nest.

>>  From an implementation standpoint: I'm already running out of ephemeral port
>> space. I *do Not* want to use more connections.
> It's certainly what browsers do already, and have for many years.
> What's the problem?

As we middleware authors put forward in the initial arguments against 
using SPDY /w TLS as a basis for HTTP/2. Browsers are not getting 
anywhere close to their TCP resource limits. Proxies and servers face 
*massively* higher port counts than individual browsers (ratios in the 
order of >20000:1 is normal). Just because browsers can consume more 
resources does not mean the other end of the connection can spare them.