Re: [hybi] Call for interest: multiplexing dedicated for WebSocket

Tobias Oberstein <> Wed, 22 May 2013 20:44 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 9A4F811E8106 for <>; Wed, 22 May 2013 13:44:38 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.284
X-Spam-Status: No, score=-2.284 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, SARE_MILLIONSOF=0.315]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id JAm6HKdKRuWK for <>; Wed, 22 May 2013 13:44:33 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 57ABA21F961B for <>; Wed, 22 May 2013 13:44:33 -0700 (PDT)
Received: from [] ( by ( with Microsoft SMTP Server (TLS) id; Wed, 22 May 2013 13:44:32 -0700
Message-ID: <>
Date: Wed, 22 May 2013 22:44:29 +0200
From: Tobias Oberstein <>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6
MIME-Version: 1.0
To: "John A. Tamplin" <>
References: <> <> <> <007501ce56f0$67f74080$37e5c180$> <> <> <> <> <>
In-Reply-To: <>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Cc: "" <>
Subject: Re: [hybi] Call for interest: multiplexing dedicated for WebSocket
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Server-Initiated HTTP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 22 May 2013 20:44:38 -0000

Am 22.05.2013 20:32, schrieb John A. Tamplin:
> On Wed, May 22, 2013 at 1:39 PM, Tobias Oberstein
> < <>> wrote:
>     Regarding implementing flow control above TCP (and below app), I
>     found this:
>     "Multiplexing on top of TCP in overall fails to deliver the
>     advantages it is assumed to provide."
>     Source:
>     <>
>     Do you have an opinion on that?
> One thing he doesn't address are resources used to keep lots of
> connections open, both at the OS level and in intermediate network nodes
> (such as load balancers -- try running tons of long-lived connections
> through an F5 for example).  Also, he doesn't address the interaction of

He does not address intermediaries, yes. But he does address memory 
consumption at endpoints: when running multiple TCPs between a given 
pair of endpoints, send/receive buffers reside in kernel space, whereas 
multiplexing over a single TCP requires send/receive buffers (per 
stream) in user space (to avoid TCP pushback). Hence he calls it a draw.

However, thinking about it: having send/receive buffers in kernel space 
requires one to decide how big buffers should be for the complete 
machine (TCP send / receive buffer size is usually a kernel tunable).

Having those buffers in user space (in app) allows one to tune the 
buffer size _per stream_. Thus, we can have large buffers for streams 
that carry mass data, and small buffers for chatty streams. Thats a real 

Rgd load balancers: yep, L7 LBs will have a problem with massive numbers 
of connections. But L4 LBs that e.g. distribute connections to backend 
cluster nodes based on consistent hash of source IP not .. since they 
dont need to maintain state per connection. A L4 LB should be able to 
scale to millions of TCPs.

> TCP slow start with lots of new connections.  I also don't believe the
> equivalent of a full TCP handshake is required to allocate buffer space.

So to make the proposed advantage explicit: the per-stream handshaking 
that happens with mux consumes less resources than full TCP handshakes.

>     More important: what problem exactly does multiplexing solve? The
>     current mux RFC mentions "scalability", but doesn't go into detail ..
> Here is a use-case I envision:  Imagine GMail, G+ sandbar, etc all use
> WebSockets heavily.  Many users may have multiple copies of such apps
> running in tabs or windows in the browser, and some of these are going
> to be separate (such as GMail and the G+ sandbar probably are separate
> apps running on the same page and maintain their own connections to
> their backends).  Currently, that would mean you have a ton of TCP
> connections open to the server.  Servers don't handle tons of long-lived

How about having a shared Web worker in the background that handles all 
messaging for the app over a single WS?

> connections very well (or intermediate load balancers), so you would

Async servers running on top of modern APIs like epoll/kqueue/IOCP have 
no issue with massive numbers of long-lived connections. I have run 180k 
WebSocket connections on a single-threaded server running on a 2 core 
4GB Ram VM (with small TCP send/receive buffers configured).

> greatly prefer to have one connection per browser.  WS MUX lets you do
> that, by having the browser aggregate connections to the same backend.
>   You could do some of it at the app level, but it is going to be
> complicated and in many cases these apps won't all be from the same
> vendor (for example, some widget on the page that uses the GData API)
> and will have separate deploy schedules.

I agree, MUX at app level is probably less than ideal.

> --
> John A. Tamplin