Re: [TLS] More compatibility measurement results

[Ilari Liusvaara <[e-mail]>]

On Fri, Dec 22, 2017 at 12:00:12PM -0800, Eric Rescorla wrote:
> Hi folks,
> 
> Here are the results of our experiment with Firefox Nightly (draft-22)
> against Facebook.
> 
> 
> RESULTS
> 
> 37716 clients started the experiment and 37430 completed it (99.2%).
> 
> The results are:
> 
>                                     Success         Fail         Rate
>                       fb-tls12        35034         1176     0.032477
>              fb-tls13-draft-22        34960         1250     0.034521
>        fb-tls13-draft22-compat        35037         1173     0.032394
> 
> None of these differences are statistically significant (in the second
> data set, the p value for 1.2 versus -22 is .13), but this all seems
> consistent with saying that that -22 compat mode isn't significantly
> worse than TLS 1.2 and that normal -22 may be somewhat worse
> (unfortunately, we don't have -18 in this experiment).
>
> Taken together with the results David has reported and our previously
> reported Beta results, this seems fairly encouraging. We'll probably
> let the Nightly experiment run a little longer to see if we hit
> significance,
> but after that will start looking at a rollout of -22 to Release.

~3.25% baseline failure rate? That sounds quite high. ~0.2% above-
baseline failure rate for non-compat? That sound fairly low, but
there have been improvements here that could have caused substantial
decrease.

I wonder if the high baseline failure rate is due to high amount of
blocking of the test server. And unfortunately, the places that
blocked the test server are some of the most interesting when it comes
to the compatibility.

However, the results do establish that the incremential failure rates
in open environments (anything that blocks the testserver very probably
is not open environment) are low enough to proceed with.


-Ilari