From: Peter Thorson <webmaster@zaphoyd.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Message-Id: <FD138330-7D7E-4450-B4F5-64551F92F26D@zaphoyd.com>
Date: Tue, 15 Oct 2013 13:34:13 -0500
To: "hybi@ietf.org" <hybi@ietf.org>
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: [hybi] permessage-deflate performance tuning statistics
Precedence: list

Hi all,

I've been doing a bit of research and testing on the compression =
performance and memory usage of permessage-deflate on WebSocket like =
workloads. I plan to write about this with more final numbers once the =
spec is official, but some of the intermediate results and tools may be =
of interest to this group during the standardization process so I'm =
sharing some of those notes here.

Some highlights:
- Permessage-deflate offers significant bandwidth savings.
- How the extension performs depends greatly on the type of data that is =
being compressed and the compression settings given to deflate.
- Settings that work well for HTTP and the zlib/permessage-deflate =
defaults are inefficient for some common WebSocket workflows.
- The two parameters presently in the draft specification both provide =
significant and meaningful options for tuning compression performance =
for those workflows. Implementations (especially browsers) are greatly =
encouraged to support all options.

Details & Methods:

My goal is to explore a number of the settings offered by deflate and =
determine what effect they have on compression performance, as well as =
CPU/memory usage. To this end I have written a tool =
(https://github.com/zaphoyd/ws-pmce-stats) that will produce a report of =
compression related statistics when fed a transcript of messages.

The first workflow I have explored in detail is a WebSocket service that =
uses a JSON based protocol to deliver short streaming updates. My some =
of my sample data is present in the datasets folder of the above git =
repository. Some examples include a mock chat service with data seeded =
from publicly logged mediawiki IRC channel and a mock stock ticker =
service with data seeded with historical stock quote data.

I explored the effects of the context takeover and window bits settings =
from the permessage-deflate draft spec as well as a few zlib settings =
that can be unilaterally specified without any extension negotiation. =
Some, but not all of these, are presently exposed in higher level =
languages that use zlib as their underlying compression library. I =
looked at two of these settings in particular, the "Compression Level" =
and the "Memory Level". The former affects speed vs compression ratio, =
the latter memory usage vs compression ratio.

Preliminary results for the JSON short message service workflow:

Context Takeover
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Allowing context takeover drastically improves compression ratios. With =
other stats at defaults, no_context_takeover achieves a compression =
ratio of 0.84, with takeover 0.30. This is a significant gain. Note: =
this gain comes at a fairly high cost. Enabling context takeover =
requires a separate context to be maintained for every connection, =
rather than fixed number for all connections.

Window Bits
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Window bits has a sizable but well distributed effect on ratios. It has =
a significant effect on memory usage though. With all other stats at =
defaults:
window bits =3D compression ratio / buffer size per connection
08 =3D 0.510 / 1+128=3D129KiB
09 =3D 0.510 / 2+128=3D130KiB
10 =3D 0.435 / 4+128=3D132KiB
11 =3D 0.384 / 8+128=3D136KiB
12 =3D 0.353 / 16+128=3D144KiB
13 =3D 0.330 / 32+128=3D160KiB
14 =3D 0.315 / 64+128=3D192KiB
15 =3D 0.304 / 128+128=3D256KiB
Reducing window bits from the default (15) to 11 provides an 8% =
reduction in compression but nearly a 50% savings in per connection =
memory usage. Reduced window bits to very small values (8-9) also =
increases compression runtime by 40-50%. 10 Is less slow, 11+ appear to =
all be about the same speed.

Compression Level
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Compression level does not have a material impact on performance or =
ratios for this workflow.

Memory Level
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Memory level does not have a significant impact on compression ratios. A =
value of 9 produces the ratio 0.304 and a value of 1 produces the ratio =
0.307. It does affect memory usage and compression speed however:
mem_level value =3D runtime / memory usage
1 =3D 13.05ms / 128+1=3D129KiB
2 =3D 10.41ms / 128+2=3D130KiB
3 =3D 10.15ms / 128+4=3D132KiB
4 =3D 8.18ms / 128+8=3D136KiB
5 =3D 7.63ms / 128+16=3D144KiB
6 =3D 7.69ms / 128+32=3D160KiB
7 =3D 7.92ms / 128+64=3D192KiB
8 =3D 7.69ms / 128+128=3D256KiB
9 =3D 7.84ms / 128+256=3D386KiB

All of the stats above show the effects of changing one parameter in =
isolation. Additional gains, especially with respect to memory usage per =
connection can be had by combinations of parameters. Many of the speed, =
compression, and memory effects of parameters are dependent on each =
other. Two nice balances of all factors for the JSON short message =
service data set (vs defaults) are something like..

context-takeover=3Don
window bits=3D11
memory level=3D4
This provides memory usage of 16KiB/connection vs 256KiB, has no runtime =
speed penalty, and achieves a 0.384 vs 0.304 compression ratio.

context-takeover=3Don
window bits=3D11
memory level=3D1
This provides memory usage of 5KiB/connection vs 256KiB, runs ~15% =
slower, and achieves a similar 0.385 vs 0.304 compression ratio.

The ws-pmce-stats tool can help you plug in values and get a sense for =
which combinations of settings are optimal for your traffic mix. In =
general I have found that the shorter your messages, the less you =
benefit from high window bit and memory level values. If you routinely =
send WebSocket messages with payloads in the high hundreds of KB or MBs =
you will benefit from higher values for memory level and window bits. If =
you have extremely limited memory, no context takeover will allow fixed =
memory usage for all connections. Its price is heavy for small messages =
& JSON protocols, but less problematic for large ones. I've found that =
11 window bits and compression memory level 4 is still quite effective =
even up to message payloads of ~200KB.

I'd love to hear any feedback anyone has about the methods or results. I =
am particularly interested in collecting more sample WebSocket =
workflows. I haven't run any numbers for binary connections yet. I'd =
love to hear details about other workflows that might have different =
properties than the ones studied here so far, especially if you have =
sample transcripts.

I'd also be interested in any feedback on the ws-pmce-stats program. Is =
something like this useful to anyone else? It meets my needs right now, =
but I have a few ideas of how to expand it (binary message support, =
other compression algorithms, machine readable output) if it sounds =
useful to others.=