[TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment

Eric Rescorla <ekr@rtfm.com> Tue, 05 December 2017 21:36 UTC

MIME-Version: 1.0
From: Eric Rescorla <ekr@rtfm.com>
Date: Tue, 05 Dec 2017 13:35:42 -0800
Message-ID: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
To: "tls@ietf.org" <tls@ietf.org>
Content-Type: multipart/alternative; boundary="f403045da8425219f5055f9ea0c2"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/RBp0X-OWNuWXugFJRV7c_hIU0dI>
Subject: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment
Precedence: list

Hi folks,

I now have some preliminary numbers to share with the group based on
our Firefox experiments. The executive summary is that our data
confirms Google's results. More detail below.

EXPERIMENTAL DESIGN
This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://mail.google.com/robots.txt) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-18
- TLS 1.3 draft-18 with (approximately) PR#1092 ("7e02")

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.

RESULTS
This experiment was run on a 2% sample of the Firefox Beta population
who have locale set to en-US, which we selected because of very
high GMail blocking rates in some locales, which is a potential
confounding factor. The experimen started 11/27 and has been running
through today.

This gave us an initial population of 161578, of whom 160809 (99.5%
completed the experiment and reported results). This produced the
following results:

Success Failure Fail Rate
--------------------------------------------------------
TLS 1.2 158260 2549 .0158
TLS 1.3-18 158194 4743 .0291
TLS 1.3-Experiment 158194 2615 .0163

For the statistics minded, the difference between -18 and 1.2 is
significant at p < .001 and the 95% confidence interval of the failure
rate difference is .0122-.0143 (using R's prop.test). There is no
significant difference between 1.2 and 1.3-experiment (p = .36).

We've got a -22 experiment in flight now, but it will only be on
Nightly, so this is probably the strongest data we will have for
a while.

-Ekr

ADDITIONAL DETAILS
The relevant NSS version:
https://dxr.mozilla.org/mozilla-beta/source/security/nss/lib/ssl
Experimental code:
https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-ghack
iPython Notebook with analysis:
https://gist.github.com/ekr/598208b5399faf303453b10cb11647bf

[TLS] Preliminary data on Firefox TLS 1.3 Middleb… Eric Rescorla
Re: [TLS] Preliminary data on Firefox TLS 1.3 Mid… Eric Rescorla