Re: [hybi] Experiment comparing Upgrade and CONNECT handshakes

[Adam Barth <[e-mail]>]

On Fri, Dec 3, 2010 at 4:26 PM, Bjoern Hoehrmann <[e-mail]> wrote:
> * Adam Barth wrote:
>>David Huang, Eric Chen, Eric Rescorla, Collin Jackson, and I have been
>>experimenting with the security of the Upgrade-based and CONNECT-based
>>WebSocket handshakes.  Please find a paper detailing our findings at
>>this location:
>>
>>http://www.adambarth.com/experimental/websocket.pdf
>
> It would have been helpful if the key metrics had been summarized in a
> simple tabular form for easy comparison (so one would not have to com-
> pare an absolute X with a relative "all but two"), but from my cursory
> reading this does not seem helpful in deciding which handshake is best.
>
> For instance, the success rate of cache poisoning is not helpful as
> they apparently compare as 8 vs 0 of about 50 000 attempts. That could
> be chance,

We didn't report the p-value for these observations in the paper, but
running a quick chi-square puts the p-value of these observations
being the same distribution around 0.0047, which is certainly
statistically significant.

> that could be a result due to the rich media banner that
> may not be supported in certain corporate or mobile settings, it could
> be due to mistakes in the experiment, and so on.

As with any experiment, there are factors we can't control.  However,
we do the best we can, and science moves forward.

> If tomorrow someone
> says they repeated the experiment and got 6 vs 3 I would find that en-
> tirely plausible, and if then someone comes along saying it's 3 vs 6
> for them, I would find that plausible too, leaving no clear winner.

I would be surprised if someone made those observations, purely on
mathematical grounds.  Another thing that's not reported in the paper
is that we did a series of pilot studies to work out the bugs in our
experiment.  Our observations were fairly stable across all these
iterations of the experiment.  Of course, reasoning statistically
about the pilot study is tricky due to post-hoc issues, which is why
we only report the final data in the paper, which we can confidently
analyze pre-hoc.

> Could you summarize the key metrics in a tabular form so I can make
> sure I haven't missed or misread any of them (between the Upgrade and
> the CONNECT handshake primarily)?

Sure.  We show that for $100, an attacker is able to succeed in
mounting 8 cache poisoning attacks against the Upgrade handshake.
Note that this value is somewhat of a lower bound because our attacks
are completely untargeted.  By targeting the attack, an attacker can
likely be even more cost-efficient.  By contrast, we never observed a
single successful cache poisoning attack against the CONNECT handshake
despite 54,534 attempts.

Adam