[TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment

Eric Rescorla <ekr@rtfm.com> Tue, 05 December 2017 21:36 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E1D7F12773A for <tls@ietfa.amsl.com>; Tue, 5 Dec 2017 13:36:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 84KbXK-YPgPX for <tls@ietfa.amsl.com>; Tue, 5 Dec 2017 13:36:24 -0800 (PST)
Received: from mail-yb0-x22b.google.com (mail-yb0-x22b.google.com [IPv6:2607:f8b0:4002:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id ED9061267BB for <tls@ietf.org>; Tue, 5 Dec 2017 13:36:23 -0800 (PST)
Received: by mail-yb0-x22b.google.com with SMTP id v12so764096ybj.5 for <tls@ietf.org>; Tue, 05 Dec 2017 13:36:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=14+ilvUq2CKNIAX18KJKfuzLbSg4tqBpQ0lsmzqLvDE=; b=Mg2o4VWxUAJrRdXnnXmhst4hydEg3KgLytGxTwlKd6TH4Uk58OGb8KVpiBfhJeXfpz bKHH3MwZqvOYK59o6ALG0o8qql4jxKGtM5QKCBU8/mGbfyTAnDCqo+bEFkMxsje+m0AP YHMhbfDjkhg7TbpGdMtTfygoX0GT644i7JwYRUJyuFloBBgcCfg2W31T4dnKTHzQRKqz kbn2BpLICu04jeDs7KOS1ps70a3PoiLIbaUY3RdgyBhGfKdWa5NyDQLMBK7B5zeBmrL+ qWZxo02nNSjDmMOIXkVydnV0QvY5lEHKSYt/YhH0czKUS95gaXOwKCJbhKxTuEGmZOkz Irvg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=14+ilvUq2CKNIAX18KJKfuzLbSg4tqBpQ0lsmzqLvDE=; b=aVHvJKh1mLBmjgGeX5VBjLyrG4aAYgK3PZtoSiAUvuW5ZWoKcW6xd0VHs0t4RqN7/w Lpze1zNr1h9ZnqhweywydOZ8yBCi16X58MnyWBnMvNaWH8HZEFWbz2q5bo0dLF+K6iTW 0xrTsWeQ4NAtfSqsm4upP36zHgVnCRaspIqa65iKTS/L5H6PCWJYwhxLthKXaCMjivEk Fe/POYesrE8A2z7t+AH/Q3ETpr1KOYLYar2IHLdcWJpduF0jWl109pxE0S1/iN4KCSBU 4LP6cWHgII48K23zZ47IF3Ub160/V1sWQsjO6/T3us5ZfX67teQMRoGVklK3D7L+i8Ut /W0w==
X-Gm-Message-State: AJaThX6P8mr0AQLXtmq7JAdbPbPhSOGL4uCyyuZrjemkGb//z2onlS3v 7q0grL9k5/ozbWe2MDGqE+8tuGJ4Dg246VESMTbfpzouorQ=
X-Google-Smtp-Source: AGs4zMZL2H4lL0kGF0w30OUFxd8dV3bSMp7s3zFF6kzYfPFFQYo8yDtrOeZ09m3Nk2gt5+p87lUC76K5/LBIA9TI4d8=
X-Received: by 10.37.246.39 with SMTP id t39mr13891883ybd.497.1512509782754; Tue, 05 Dec 2017 13:36:22 -0800 (PST)
MIME-Version: 1.0
Received: by 10.129.123.132 with HTTP; Tue, 5 Dec 2017 13:35:42 -0800 (PST)
From: Eric Rescorla <ekr@rtfm.com>
Date: Tue, 5 Dec 2017 13:35:42 -0800
Message-ID: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
To: "tls@ietf.org" <tls@ietf.org>
Content-Type: multipart/alternative; boundary="f403045da8425219f5055f9ea0c2"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/RBp0X-OWNuWXugFJRV7c_hIU0dI>
Subject: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Dec 2017 21:36:26 -0000

Hi folks,

I now have some preliminary numbers to share with the group based on
our Firefox experiments. The executive summary is that our data
confirms Google's results. More detail below.


EXPERIMENTAL DESIGN
This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://mail.google.com/robots.txt) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-18
- TLS 1.3 draft-18 with (approximately) PR#1092 ("7e02")

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.


RESULTS
This experiment was run on a 2% sample of the Firefox Beta population
who have locale set to en-US, which we selected because of very
high GMail blocking rates in some locales, which is a potential
confounding factor. The experimen started 11/27 and has been running
through today.

This gave us an initial population of 161578, of whom 160809 (99.5%
completed the experiment and reported results). This produced the
following results:

                     Success      Failure      Fail Rate
--------------------------------------------------------
TLS 1.2               158260         2549          .0158
TLS 1.3-18            158194         4743          .0291
TLS 1.3-Experiment    158194         2615          .0163

For the statistics minded, the difference between -18 and 1.2 is
significant at p < .001 and the 95% confidence interval of the failure
rate difference is .0122-.0143 (using R's prop.test). There is no
significant difference between 1.2 and 1.3-experiment (p = .36).

We've got a -22 experiment in flight now, but it will only be on
Nightly, so this is probably the strongest data we will have for
a while.

-Ekr


ADDITIONAL DETAILS
The relevant NSS version:
https://dxr.mozilla.org/mozilla-beta/source/security/nss/lib/ssl
Experimental code:
https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-ghack
iPython Notebook with analysis:
https://gist.github.com/ekr/598208b5399faf303453b10cb11647bf