Re: [hybi] Call for interest: multiplexing dedicated for WebSocket

[Tobias Oberstein <[e-mail]>]

>>Having those buffers in user space (in app) allows one to tune the buffer size _per stream_. Thus, we can have large buffers for streams that carry mass data, and small buffers for chatty streams. Thats a real >>advantage.

>Exactly.

In the meantime, I have read more about modern TCP/IP stacks.

E.g. beginning with FreeBSD 7, the TCP/IP stack auto-tunes the TCP send/receive buffer sizes _per socket_. This is controlled via:

net.inet.tcp.sendbuf_auto=1         # TCP send buffer size autotuning (default: on)
net.inet.tcp.sendbuf_inc=16384      # TCP send buffer autotuning step size
net.inet.tcp.recvbuf_auto=1         # TCP receive buffer size autotuning (default: on)
net.inet.tcp.recvbuf_inc=524288     # TCP receive buffer autotuning step size
net.inet.tcp.sendbuf_max=16777216   # TCP send buffer maximum size (tune up for Fat-Long-Pipes)
net.inet.tcp.recvbuf_max=16777216   # TCP receive buffer maximum size (tune up for Fat-Long-Pipes)

"""
TCP Autotuning

Beginning with Linux 2.6, Mac OSX 10.5, Windows Vista, and FreeBSD 7.0, both sender and receiver autotuning became available, eliminating the need to set the TCP send and receive buffers by hand for each path. However the maximum buffer sizes are still too small for many high-speed network path, and must be increased as described on the pages for each operating system.
"""
http://fasterdata.es.net/host-tuning/background/

Apart from autotuning, there is also the option to control send/receive buffer sizes _per socket_ using setsockopt() with SO_SNDBUF / SO_RCVBUF.

===

Given above, how would the following 2 approaches compare?

a)
- Client 1 <=> Server 1 connected with 2 TCP connections.
- TCP autotuning on both client and server OR manually set buffer sizes: small for TCP-1, and large for TCP-2
- First TCP _only_ carries "chatty" traffic, and second TCP carries "mass data" traffic.

b)
- Client 1 <=> Server 1 connected with 1 TCP connection
- TCP autotuning OR large buffers set system wide
- multiplexing done over the single TCP that carries both "chatty" and "mass data" traffic and
- the multiplexing is further optimizing buffer size (app level) and does priorization of traffic (chatty = high-prio, mass = low-prio):

"Compare": I am mostly interested in the end-user experience:

i) Is the chatty traffic still low-latency, even in the presence of concurrent mass-data traffic?
ii) Is the throughput of mass-data traffic concurrently to the chatty traffic still (nearly) as high as when only doing mass data transfer?
 
>>How about having a shared Web worker in the background that handles all messaging for the app over a single WS?
>One restriction of Web Worker is that we cannot share a worker between webapps with different origins while WebSocket can connect to a host different from its origin.

Ah, right. Same origin policy. Thus, with WS-MUX, there is _more_ potentially TCP sharing than whats possible with shared WebWorkers.

However, for a given single app (from a single origin) running in multiple browser tabs, the shared WebWorker seems not so bad. Works today. No need for WS-MUX or HTTP 2.0 MUX + WS over HTTP 2.0.

>For load balancer vendors, it's better that multiplexing is standardized than each service provider develops its own.

I don't have experience running G scale infrastructure. However, I'd like to understand why LBs need to have a look into WS at all .. why not Layer-4 LB? Doing balancing based on hash of source IP?

>Basically multiplexing should be done as close to network as possible. If it's done in application layer, for example when compression is used, such a load balancer need to decode the compression to see the >multiplexing information. That doesn't make them happy.

This seems to assume that 

- either the LB is done above layer 4 and/or
- the logical WS connections contained in the physical WS would need to be balanced to _different_ backend nodes.

>As Google, our motivation of mux is to make sure that total meaningful traffic per server cost (memory, port, machine, etc.) small enough compared to HTTP/Comet approach.
 
I see. For me, I am mostly interested in the use case: 1 app doing both chatty and mass-data coms.