[TLS] More compatibility measurement results

[Eric Rescorla <[e-mail]>]

Hi folks,

Here are the results of our experiment with Firefox Nightly (draft-22)
against Facebook.

EXPERIMENTAL DESIGN
This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://www.tls13.facebook.com/) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-22
- TLS 1.3 draft-22 in compat mode.

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.


RESULTS
This experiment was run on a 40% sample of the Firefox Nightly population
who have locale set to en-US. The data below is taken from the period
20171216 to 20171222. There's a bit of contamination in the targeting
because we temporarily failed to filter on "en-US", but that should
mostly only affect the first day.

37716 clients started the experiment and 37430 completed it (99.2%).

The results are:

                                    Success         Fail         Rate
                      fb-tls12        35615         1815     0.048491
             fb-tls13-draft-22        35552         1878     0.050174
       fb-tls13-draft22-compat        35630         1800     0.048090

The overall failure numbers here are a lot higher than with our Beta
experiment, which may be a result of different targeting on Nightly
versus Beta. In particular, I'm still seeing a lot of data from China
and Vietnam, which seem to have high blocking rates in general (i.e.,
not just for TLS 1.3). If I restrict to non China and non-Vietnam, we
get:

                                    Success         Fail         Rate
                      fb-tls12        35034         1176     0.032477
             fb-tls13-draft-22        34960         1250     0.034521
       fb-tls13-draft22-compat        35037         1173     0.032394

None of these differences are statistically significant (in the second
data set, the p value for 1.2 versus -22 is .13), but this all seems
consistent with saying that that -22 compat mode isn't significantly
worse than TLS 1.2 and that normal -22 may be somewhat worse
(unfortunately, we don't have -18 in this experiment).

Taken together with the results David has reported and our previously
reported Beta results, this seems fairly encouraging. We'll probably
let the Nightly experiment run a little longer to see if we hit
significance,
but after that will start looking at a rollout of -22 to Release.

-Ekr


ADDITIONAL DETAILS
Experimental code:
https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-draft22