RE: Choosing a header compression algorithm

[RUELLAN Herve <[e-mail]>]

I think it would be good to move this from the compressors to the streamifier. In addition, it would be interesting to look at a more realistic streamifier that could for example unshard hosts (expecting that HTTP/2.0 will remove the sharding currently done by server developers).

Hervé.

From: Roberto Peon [mailto:[e-mail]]
Sent: vendredi 22 mars 2013 18:46
To: RUELLAN Herve
Cc: Mark Nottingham; [e-mail] Group
Subject: Re: Choosing a header compression algorithm

Correct, delta2 uses a header group per host, however, the real benefits of this don't show up with the current streamifier, which removes the intermingling of requests to different hosts (which is a bit unrealistic, but does makes the graphs easier to parse visually).
This can end up really mattering to the compression in the cases where there is some ping-ponging between different host headers.

-=R

On Fri, Mar 22, 2013 at 9:01 AM, RUELLAN Herve <[e-mail]<mailto:[e-mail]>> wrote:
The results for HeaderDiff were not at all those I excepted.

I traced the difference to the number of compressor instances used to compress the sample files. The framework uses only one instance for all the files, while internally I used one instance for each file.

I noticed that the delta2 compressor seems to use one compression context for each host. I think this is the correct behavior to obtain realistic results. I will therefore update the HeaderDiff codec to create one compression context per host.

Hervé.

> -----Original Message-----
> From: Mark Nottingham [mailto:[e-mail]<mailto:[e-mail]>]
> Sent: vendredi 22 mars 2013 07:32
> To: [e-mail]<mailto:[e-mail]> Group
> Subject: Re: Choosing a header compression algorithm
>
>
> On 21/03/2013, at 6:11 PM, Mark Nottingham <[e-mail]<mailto:[e-mail]>> wrote:
>
> > 1) We need to do apples-to-apples comparison of these compressors to
> see how they behave under a range of constraints (especially, memory).
>
> To this point, a few runs across the header corpus that I collected
> (http_samples/mnot/*.har), using the current implementations (again, we
> need to verify these are honouring these limits correctly, and I believe
> headerdiff is going to be updated):
>
> 16384 Byte Buffer
> ----------------------
>
> * TOTAL: 5948 req messages
>                                         size  time | ratio min   max   std
>                          http1     3,460,925  0.22 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=16384)       630,668 15.89 | 0.18  0.03  0.76  0.11
>      headerdiff (buffer=16384)     1,699,918  9.65 | 0.49  0.03  0.90  0.21
>
> * TOTAL: 5948 res messages
>                                         size  time | ratio min   max   std
>                          http1     2,186,162  0.26 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=16384)       575,345 24.39 | 0.26  0.02  1.06  0.12
>      headerdiff (buffer=16384)       877,106 24.76 | 0.40  0.04  0.93  0.14
>
> [ Note that the efficiency actually goes *down* on headerdiff for requests
> when the buffer is larger here; Herve, any idea what's happening there? ]
>
> 8192 Byte Buffer
> ----------------------
>
> * TOTAL: 5948 req messages
>                                        size  time | ratio min   max   std
>                         http1     3,460,925  0.24 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=8192)       646,431 15.78 | 0.19  0.03  0.76  0.12
>      headerdiff (buffer=8192)     1,378,622  6.95 | 0.40  0.03  0.92  0.20
>
> * TOTAL: 5948 res messages
>                                        size  time | ratio min   max   std
>                         http1     2,186,162  0.43 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=8192)       592,260 20.28 | 0.27  0.02  1.06  0.12
>      headerdiff (buffer=8192)       914,362 15.55 | 0.42  0.05  0.93  0.14
>
>
> 4096 Byte Buffer
> ----------------------
>
> * TOTAL: 5948 req messages
>                                        size  time | ratio min   max   std
>                         http1     3,460,925  0.18 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=4096)       665,642 13.35 | 0.19  0.03  0.77  0.12
>      headerdiff (buffer=4096)     1,406,429  3.92 | 0.41  0.03  0.88  0.18
>
> * TOTAL: 5948 res messages
>                                        size  time | ratio min   max   std
>                         http1     2,186,162  0.21 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=4096)       610,764 15.48 | 0.28  0.02  1.06  0.13
>      headerdiff (buffer=4096)     1,009,185  7.58 | 0.46  0.05  0.93  0.13
>
> 2048 Byte Buffer
> ----------------------
>
> * TOTAL: 5948 req messages
>                                        size  time | ratio min   max   std
>                         http1     3,460,925  0.18 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=2048)       692,579 11.66 | 0.20  0.03  0.77  0.12
>      headerdiff (buffer=2048)     1,632,783  2.94 | 0.47  0.03  0.90  0.19
>
> * TOTAL: 5948 res messages
>                                        size  time | ratio min   max   std
>                         http1     2,186,162  0.20 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=2048)       633,785 12.47 | 0.29  0.02  1.22  0.13
>      headerdiff (buffer=2048)     1,088,829  4.75 | 0.50  0.05  0.93  0.13
>
> 1024 Byte Buffer
> ----------------------
>
> * TOTAL: 5948 req messages
>                                        size  time | ratio min   max   std
>                         http1     3,460,925  0.22 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=1024)       752,380 11.92 | 0.22  0.03  0.84  0.13
>      headerdiff (buffer=1024)     1,700,325  1.82 | 0.49  0.03  0.91  0.19
>
> * TOTAL: 5948 res messages
>                                        size  time | ratio min   max   std
>                         http1     2,186,162  0.23 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=1024)       671,975 11.42 | 0.31  0.02  1.22  0.13
>      headerdiff (buffer=1024)     1,118,599  3.53 | 0.51  0.05  0.93  0.13
>
> 512 Byte Buffer
> ---------------------
>
> * TOTAL: 5948 req messages
>                                       size  time | ratio min   max   std
>                        http1     3,460,925  0.17 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=512)     1,058,090 14.23 | 0.31  0.04  0.83  0.15
>      headerdiff (buffer=512)     2,120,833  1.43 | 0.61  0.03  0.93  0.16
>
> * TOTAL: 5948 res messages
>                                       size  time | ratio min   max   std
>                        http1     2,186,162  0.14 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=512)       721,525 12.22 | 0.33  0.03  1.22  0.13
>      headerdiff (buffer=512)     1,199,608  2.09 | 0.55  0.05  0.93  0.12
>
> 256 Byte Buffer
> ---------------------
>
> * TOTAL: 5948 req messages
>                                       size  time | ratio min   max   std
>                        http1     3,460,925  0.18 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=256)     1,517,520 18.62 | 0.44  0.04  0.84  0.19
>      headerdiff (buffer=256)     2,444,376  1.32 | 0.71  0.08  0.95  0.13
>
> * TOTAL: 5948 res messages
>                                       size  time | ratio min   max   std
>                        http1     2,186,162  0.23 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=256)       856,632 13.21 | 0.39  0.04  1.22  0.14
>      headerdiff (buffer=256)     1,201,814  1.65 | 0.55  0.05  0.93  0.12
>
> 32 Byte Buffer
> -------------------
>
> * TOTAL: 5948 req messages
>                                      size  time | ratio min   max   std
>                       http1     3,460,925  0.20 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=32)     2,357,457 28.18 | 0.68  0.51  0.84  0.05
>      headerdiff (buffer=32)     2,791,039  1.16 | 0.81  0.58  0.96  0.08
>
> * TOTAL: 5948 res messages
>                                      size  time | ratio min   max   std
>                       http1     2,186,162  0.21 | 1.00  1.00  1.00  0.00
>   delta2 (max_byte_size=32)     1,197,019 17.92 | 0.55  0.16  0.89  0.07
>      headerdiff (buffer=32)     1,434,721  1.32 | 0.66  0.19  0.95  0.07
>
> [ smaller buffer sizes caused errors in delta2 (at least) ]
>
> --
> Mark Nottingham   http://www.mnot.net/
>
>
>