[TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment

Eric Rescorla <ekr@rtfm.com> Tue, 05 December 2017 21:36 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id E1D7F12773A for <tls@ietfa.amsl.com>; Tue, 5 Dec 2017 13:36:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id 84KbXK-YPgPX for <tls@ietfa.amsl.com>; Tue, 5 Dec 2017 13:36:24 -0800 (PST)
Received: from mail-yb0-x22b.google.com (mail-yb0-x22b.google.com [IPv6:2607:f8b0:4002:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id ED9061267BB for <tls@ietf.org>; Tue, 5 Dec 2017 13:36:23 -0800 (PST)
Received: by mail-yb0-x22b.google.com with SMTP id v12so764096ybj.5 for <tls@ietf.org>; Tue, 05 Dec 2017 13:36:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=14+ilvUq2CKNIAX18KJKfuzLbSg4tqBpQ0lsmzqLvDE=; b=Mg2o4VWxUAJrRdXnnXmhst4hydEg3KgLytGxTwlKd6TH4Uk58OGb8KVpiBfhJeXfpz bKHH3MwZqvOYK59o6ALG0o8qql4jxKGtM5QKCBU8/mGbfyTAnDCqo+bEFkMxsje+m0AP YHMhbfDjkhg7TbpGdMtTfygoX0GT644i7JwYRUJyuFloBBgcCfg2W31T4dnKTHzQRKqz kbn2BpLICu04jeDs7KOS1ps70a3PoiLIbaUY3RdgyBhGfKdWa5NyDQLMBK7B5zeBmrL+ qWZxo02nNSjDmMOIXkVydnV0QvY5lEHKSYt/YhH0czKUS95gaXOwKCJbhKxTuEGmZOkz Irvg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=14+ilvUq2CKNIAX18KJKfuzLbSg4tqBpQ0lsmzqLvDE=; b=aVHvJKh1mLBmjgGeX5VBjLyrG4aAYgK3PZtoSiAUvuW5ZWoKcW6xd0VHs0t4RqN7/w Lpze1zNr1h9ZnqhweywydOZ8yBCi16X58MnyWBnMvNaWH8HZEFWbz2q5bo0dLF+K6iTW 0xrTsWeQ4NAtfSqsm4upP36zHgVnCRaspIqa65iKTS/L5H6PCWJYwhxLthKXaCMjivEk Fe/POYesrE8A2z7t+AH/Q3ETpr1KOYLYar2IHLdcWJpduF0jWl109pxE0S1/iN4KCSBU 4LP6cWHgII48K23zZ47IF3Ub160/V1sWQsjO6/T3us5ZfX67teQMRoGVklK3D7L+i8Ut /W0w==
X-Gm-Message-State: AJaThX6P8mr0AQLXtmq7JAdbPbPhSOGL4uCyyuZrjemkGb//z2onlS3v 7q0grL9k5/ozbWe2MDGqE+8tuGJ4Dg246VESMTbfpzouorQ=
X-Google-Smtp-Source: AGs4zMZL2H4lL0kGF0w30OUFxd8dV3bSMp7s3zFF6kzYfPFFQYo8yDtrOeZ09m3Nk2gt5+p87lUC76K5/LBIA9TI4d8=
X-Received: by with SMTP id t39mr13891883ybd.497.1512509782754; Tue, 05 Dec 2017 13:36:22 -0800 (PST)
MIME-Version: 1.0
Received: by with HTTP; Tue, 5 Dec 2017 13:35:42 -0800 (PST)
From: Eric Rescorla <ekr@rtfm.com>
Date: Tue, 05 Dec 2017 13:35:42 -0800
Message-ID: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
To: "tls@ietf.org" <tls@ietf.org>
Content-Type: multipart/alternative; boundary="f403045da8425219f5055f9ea0c2"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/RBp0X-OWNuWXugFJRV7c_hIU0dI>
Subject: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Dec 2017 21:36:26 -0000

Hi folks,

I now have some preliminary numbers to share with the group based on
our Firefox experiments. The executive summary is that our data
confirms Google's results. More detail below.

This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://mail.google.com/robots.txt) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-18
- TLS 1.3 draft-18 with (approximately) PR#1092 ("7e02")

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.

This experiment was run on a 2% sample of the Firefox Beta population
who have locale set to en-US, which we selected because of very
high GMail blocking rates in some locales, which is a potential
confounding factor. The experimen started 11/27 and has been running
through today.

This gave us an initial population of 161578, of whom 160809 (99.5%
completed the experiment and reported results). This produced the
following results:

                     Success      Failure      Fail Rate
TLS 1.2               158260         2549          .0158
TLS 1.3-18            158194         4743          .0291
TLS 1.3-Experiment    158194         2615          .0163

For the statistics minded, the difference between -18 and 1.2 is
significant at p < .001 and the 95% confidence interval of the failure
rate difference is .0122-.0143 (using R's prop.test). There is no
significant difference between 1.2 and 1.3-experiment (p = .36).

We've got a -22 experiment in flight now, but it will only be on
Nightly, so this is probably the strongest data we will have for
a while.


The relevant NSS version:
Experimental code:
iPython Notebook with analysis: