Re: bohe implementation for compression tests

[James M Snell <[e-mail]>]

Just continuing my investigation of various header compression strategies
around the BOHE mechanism. In my personal github fork, I have just checked
in two bohe variations, one that implements selective-compression, the
other implements isolated-compression..

https://github.com/jasnell/compression-test/tree/master/compressor/bohe2
https://github.com/jasnell/compression-test/tree/master/compressor/bohe3

With bohe2 (selective-compression), a header block can consist of a
compressed set of headers and an uncompressed set of headers. Specific
headers such as Cookie, Set-Cookie, etc can be marked as "Do Not Compress".
These are dropped into the frame as-is and thus avoid the CRIME issue
completely. The rest of the headers are compressed using gzip using the
existing spdy3 dictionary. Obviously this is not ideal because Cookie data
then is passed around without any compression at all, making it far less
efficient than any of the other options on the table.

In bohe3 (isolated-compression), a header block can consist of two separate
compressed blocks generated using two separate stream compressor instances.
Selected headers (like Cookie) can be included in the secondary isolated
block which would never contain general user-provided header data.

**These are only experiments right now and are not intended as serious
proposals for the spec**. Isolated-compression (bohe3) does show promise
however. If we can successfully isolate potentially sensitive headers into
their own compression context generated independently of any general
user-supplied data, we can effectively short-circuit the CRIME attack by
making it impossible for an attacker to compare values based on the
compression ratio... and since it still uses gzip compression, we achieve a
generally better compression ratio overall than we get with the proposed
delta encoding. For now, tho, just consider this all just to be fodder for
discussion. There are still MANY issues with these experimental approaches
and I still need to go through delta in more detail to see if there is a
way bohe and delta can be used effectively together.

Just for example...

james-snells-macbook-pro:compression-test james$ ./compare_compressors.py
-c bohe3 -c bohe2 -c bohe -c delta -t
/Users/james/git/http_samples/mnot/amazon.com.har
732 req messages processed
             compressed | ratio min   max   std
req  bohe        26,035 | 0.13  0.03  0.68  0.08
req bohe2        44,195 | 0.23  0.07  0.71  0.13
req bohe3        30,944 | 0.16  0.05  0.74  0.08
req delta        33,955 | 0.17  0.02  0.71  0.09
req http1       195,386 | 1.00  1.00  1.00  0.00

732 res messages processed
             compressed | ratio min   max   std
res  bohe        39,525 | 0.25  0.04  0.67  0.07
res bohe2        47,157 | 0.29  0.12  0.71  0.08
res bohe3        44,843 | 0.28  0.06  0.70  0.07
res delta        44,499 | 0.28  0.02  0.65  0.09
res http1       159,968 | 1.00  1.00  1.00  0.00

- James



On Thu, Jan 10, 2013 at 11:08 AM, James M Snell <[e-mail]> wrote:

> I have an initial bohe implementation for the compression tests... it's
> very preliminary and uses the same gzip compression as the current spdy3.
> I'm going to be playing around with the delta compression mechanism as well
> and see how much of an impact that has. Initial results are very promising
> but I haven't done much debugging yet. Just wanted folks to know that this
> work was underway...
>
> https://github.com/jasnell/compression-test/tree/master/compressor/bohe
>
> Some test runs....
>
> ./compare_compressors.py -c bohe -c spdy3 -c delta
> ../http_samples/mnot/amazon.com.har
> 732 req messages processed
>              compressed | ratio min   max   std
> req  bohe        26,122 | 0.13  0.04  0.70  0.08
> req delta        33,955 | 0.17  0.02  0.71  0.09
> req http1       195,386 | 1.00  1.00  1.00  0.00
> req spdy3        27,238 | 0.14  0.04  0.71  0.08
>
> 732 res messages processed
>              compressed | ratio min   max   std
> res  bohe        39,628 | 0.25  0.04  0.66  0.07
> res delta        44,499 | 0.28  0.02  0.65  0.09
> res http1       159,968 | 1.00  1.00  1.00  0.00
> res spdy3        41,325 | 0.26  0.04  0.67  0.08
>
>
> ./compare_compressors.py -c bohe -c spdy3 -c delta
> ../http_samples/mnot/craigslist.org.har
> 66 req messages processed
>              compressed | ratio min   max   std
> req  bohe         1,948 | 0.15  0.06  0.73  0.11
> req delta         2,036 | 0.16  0.07  0.71  0.11
> req http1        12,894 | 1.00  1.00  1.00  0.00
> req spdy3         2,016 | 0.16  0.07  0.75  0.11
>
> 66 res messages processed
>              compressed | ratio min   max   std
> res  bohe         1,786 | 0.18  0.07  0.77  0.13
> res delta         2,858 | 0.28  0.08  0.69  0.12
> res http1        10,147 | 1.00  1.00  1.00  0.00
> res spdy3         1,869 | 0.18  0.09  0.78  0.13
>
>
> ./compare_compressors.py -c bohe -c spdy3 -c delta
> ../http_samples/mnot/flickr.com.har
> 438 req messages processed
>              compressed | ratio min   max   std
> req  bohe        11,988 | 0.10  0.02  0.69  0.07
> req delta        26,372 | 0.22  0.01  0.71  0.14
> req http1       121,854 | 1.00  1.00  1.00  0.00
> req spdy3        12,550 | 0.10  0.02  0.71  0.07
>
> 438 res messages processed
>              compressed | ratio min   max   std
> res  bohe        13,073 | 0.09  0.05  0.66  0.06
> res delta        25,236 | 0.18  0.02  0.70  0.11
> res http1       140,457 | 1.00  1.00  1.00  0.00
> res spdy3        14,142 | 0.10  0.05  0.66  0.06
>
>
> ./compare_compressors.py -c bohe -c spdy3 -c delta
> ../http_samples/mnot/facebook.com.har
> 234 req messages processed
>              compressed | ratio min   max   std
> req  bohe         6,091 | 0.15  0.06  0.78  0.07
> req delta         7,800 | 0.19  0.02  0.70  0.07
> req http1        41,980 | 1.00  1.00  1.00  0.00
> req spdy3         6,301 | 0.15  0.06  0.77  0.07
>
> 234 res messages processed
>              compressed | ratio min   max   std
> res  bohe         9,458 | 0.23  0.07  0.68  0.07
> res delta        12,045 | 0.30  0.13  0.60  0.08
> res http1        40,252 | 1.00  1.00  1.00  0.00
> res spdy3         9,788 | 0.24  0.07  0.69  0.07
>
>
>
>
>