Re: [hybi] preliminary WebSockets compression experiments

[Mike Belshe <[e-mail]>]

On Mon, Apr 26, 2010 at 7:05 AM, Greg Wilkins <[e-mail]> wrote:

> Mike,
>
> some slightly off topic thinking out loud on my idea that websockets
> should be able to eventually replace the framing layer of SPDY....
>
> The problem that SPDY poses for efficient compression implemented in the
> framing layer is that it's frames may contain both headers and
> a content body.
>

The most efficient compression will always come from the entity which knows
the most about what it is compressing.   Compressing headers separately from
other content compresses more efficiently than if you mix compression of
headers and data.  (Imagine a gzip buffer with 32KB of window where you
first send 500B of headers and then 50KB of data through it - by the time
your next chunk of headers come through, all state from the prior headers
has been lost.  And the data that you sent through might have been
compressed anyway!)

I welcome you to test out this theory - I've done so and I know the answer
:-)   As you can see, header-specific compression is a feature.

Also - remember the note about zlib memory footprint.  Using an 11bit window
size works just fine for headers because they are generally small, and this
keeps overall zlib memory pretty low (25K or so?).  But, if you want a
general compressor, you'll need the full 15bit compression window if you
want decent efficiency, and then you're looking at 256KB of RAM for each
zlib compression state instance.  I know, you could use a different
compressor - but most compressors will have a similar property.

The point is that a content-specific compressor will almost always
outperform a generic compressor in efficiency, speed, and footprint.



>
> The headers are very compressible, but SPDY will probably have
> already compressed them. The content body may be very compressible
> (eg HTML) or not (eg JPG).
>
> So if we were to run SPDY over websocket, then it would be unlikely
> that the compression mechanism in websocket would be used, as SPDY
> will have already compressed what is compressible.   It strikes me
> as somewhat duplicated/wasted effort to have compression in both
> layers.
>

Roberto tried to convey why optional compression is a bad thing sometimes.
 Let me use a real world example.

We've wondered here at Google why so many requests come from browsers
claiming not to support compression.  After a lot of investigation (I was
not involved with this work), we ran into client requests which contained
headers like:
     Accept-Xncoding: gzip, deflate

What is that?

Well, it turns out that some anti-virus providers didn't want to support
compression in payloads.  Doing so meant that sometimes they couldn't
inspect the payloads, and they were too lazy to implement the basic,
well-known compressors.  So these anti-virus programs watched for outbound
HTTP requests and intentionally mangled the browser's outbound
"Accept-Encoding" header.  This prevents the server from knowing the browser
supports compression, and thus the anti-virus software can look through the
result without having to do any decompress work.   But what happened here?
 The poor user in this case just had the single-worst-thing happen to his
browser - no compression at all!!!  And all because of some lazy
man-in-the-middle.

This is an example where too much flexibility over an insecure pipe bites
you.  And the best answer is to have the protocol help avoid it.  You can do
so by encrypting everything to avoid tampering, or you can avoid it by
mandating compression.  If compression really is a key feature of the
protocol, then allowing some implementations to turn it off is a bad idea.
 It will come back to haunt you.

Summary:
  * Compression is always best handled by the party that knows what is being
compressed.  (Compress at the highest layer possible)
  * Generic stream-compression of unknown content is okay, but not great.
 (because the content may well already be compressed)
  * Don't allow endpoints to not-support compression.  If they're allowed to
skip it, you end up with a critical feature missing for some poor users.
  * Don't allow intermediaries to disable your compression. This can only be
done by removing configuration flags/bits or by encrypting everything.



>
> Ideally websockets should be able to provide framing compression
> sufficient so that SPDY (or any new application layer) would not
> be required to have it's own compression mechanisms.
>

I don't buy it for the reasons above.  But if you can prove me wrong, then
great!



>
> For SPDY, the problem of using framing compression is that the
> decision to compress the headers and/or the body is separate.
>

Why is this a problem?


> So perhaps the frame batch concept in BWTP might be useful
> after all - so that a HTTP message could be a batch of a
> two frames, one (normally compressed) containing the headers
> and another (optionally compressed) containing the body.
>

Okay.  I didn't have a problem with the batch frame - it's just a different
syntax.

Mike




>
> Note also that compression efficiency means that the current
> 1 header field per frame approach in the latest BWTP draft
> is a non starter.
>
>
>
> cheers
>
>
>
>