Re: [hybi] preliminary WebSockets compression experiments

[John Tamplin <[e-mail]>]

On Fri, Apr 23, 2010 at 5:48 PM, Mike Belshe <[e-mail]> wrote:

> *Memory Consumption:*
> BTW - zlib, using the configuration you specified, might use a surprising
> amount of RAM (to the tune of ~250KB!).  I suspect you want to decrease the
> window size. Brian Olson already ran this experiment a while back with SPDY
> and his results are here.  It would be interesting to see if you reach the
> same conclusion.
>
> groups.google.com/group/spdy-dev/browse_thread/thread/dfaf498542fac792?pli=1
>
> As a result of Brian's work, SPDY decreased the windowBits from 15 to 11.
>  This reduces the memory footprint substantially, while only leading to
> modest reduction in compression ratios.
>

Thanks for the pointer -- I will try that out.


> *On Generically Compressing A Stream:*
> While we wanted to have SPDY provide required compression, in the end we
> find it likely a bad idea.  Compressing the stream is never as good as
> compressing the content.  For example, if the content can compress the
> image, there is no point in having the stream redundantly (and likely
> less-effectively) compress it.  Since most developers will get the content
> compressed correctly, that is the right layer to do so; rather than having
> the channel blindly compress what is already done.  We might revisit this
> approach if new data arises, though.
>
> SSL has taken a similar approach.  When the protocol was designed, the
> authors had the foresight to include compression negotiation in addition to
> cypher negotiation as part of the handshake.  They did this because
> compression should be applied before encryption (compressing encrypted data
> is never good).  But, what do browsers today negotiate?  The major browsers
> all advertise the empty-set of supported compression algorithms (meaning
> don't do compression).  Why?  because compressing the stream blindly when
> the application content is already compressed is inefficient.
>
> Anyway - those are two protocols which are opting not to do stream-based
> compression and instead requiring the content provider to make the choice
> (like HTTP does).  I think this matches your conclusion that you shouldn't
> just compress everything.
>

I agree with the basic premise that the application can do a better job
compressing the data than some generic algorithm -- my compression
background is chess endgame databases where application-specific compression
algorithms are much better than generic ones.  However, I don't know about
trying to implement that in JS along with a corresponding server-side
implementation.

For HTTP, most servers are configured to compress things automatically if
the client supports it, typically with a blacklist of file types not to
compress (granted, you can pre-compress static content and serve that, and
some servers may be intentionally configured to not compress other data).
 Clearly using gzip compression on HTTP traffic is generally a good thing,
and especially in the case of mobile or other low-bandwidth networks.

As the GWT Quake example shows, even data that you wouldn't think is
compressible (and isn't if you don't maintain compression state) compresses
quite well.  With the safety valve of allowing uncompressed frames even when
compression has been negotiated, I don't see how it can be a bad thing.  The
server-side API might allow a way for the particular service on the other
end of the socket to deny compression if it knows its data can't benefit,
but that is really beyond the scope of the wire format.

It would be feasible to provide an API for JS to call for compressing data
and sending it itself, but there are a couple of problems:
 - JS currently doesn't have any efficient way to manipulate binary data
 - in the common case, it just adds extra work for the developer and extra
code to download

I think that might be a good idea once the binary data problem is solved,
but I think a default case of allowing reasonable compression on the stream
automatically is a good idea.

-- 
John A. Tamplin
Software Engineer (GWT), Google