Re: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment

Eric Rescorla <ekr@rtfm.com> Tue, 05 December 2017 21:54 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 07BFA12869B for <tls@ietfa.amsl.com>; Tue, 5 Dec 2017 13:54:37 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Level:
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z2heS5zmU-_V for <tls@ietfa.amsl.com>; Tue, 5 Dec 2017 13:54:34 -0800 (PST)
Received: from mail-yw0-x22d.google.com (mail-yw0-x22d.google.com [IPv6:2607:f8b0:4002:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7813D128891 for <tls@ietf.org>; Tue, 5 Dec 2017 13:54:34 -0800 (PST)
Received: by mail-yw0-x22d.google.com with SMTP id f1so771689ywd.8 for <tls@ietf.org>; Tue, 05 Dec 2017 13:54:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=kqKKrnt+0s/YHIHzf5zcZaW7gViRTJ892l9z6k9jljc=; b=r3ejOsPZ5PA4aEkmeXe2xEqt6At0bi9o7HEU9G4BEbSEvn85R3GsrJ8HOFnOFgv1yO DWra7dyWJaP8UhQyjQnKFtAMZi1BfbDDfhK/w+nNs/oB1tB93kL6oHM91XW60xcWJNm2 /TM85qInlOuWLPUQ0m5HpbiVhO1R5jrRNfWbwOqTGB64Ixyg8Jdqqvm8vsR33vga3GAG a9MWJdBa01RKzyHbp3894ixvYuAFDpqhSpM8tOEkjzitPARBIFxjftVWpLSNiXPYPgPy Ut988/iE3me3nE4itPbmIpRqLfVQhZMgaIWKSYvPAhsUff9LvwG7Ztp3coe2V01mEy2X EFZQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=kqKKrnt+0s/YHIHzf5zcZaW7gViRTJ892l9z6k9jljc=; b=q27hE2XDEDsuNqXGooPfjWOvb9nqvOcsE7wMIfUAZjv+a6lRAFhEDzgy3s9/V8SOVv eEi7ff+0GFbsJKIs8ghcD7CzyEOlVmVqrRh9cShOoMmznHrCDnqdTlFY5qjsDCI6g1LH MNXqUrbL455tGdTnAjJtUCm+o+uLkHP9iLH3kQ1DP66pCi6/dP9hr8+GoGABqiZnrFKJ 6+/qJpG1g0pEKPeDpcwNHJDzSvvrpUKILa3JxCPcXNJNLyYUo/A1xtiJTJxV7A2j+Ht7 0eLV6IJpyMypmmh/lEK+Q2vdj8IZU8OZre0c/cM1KeNHu7EAE9ESiYHGyHBEVpKhCIRC baIQ==
X-Gm-Message-State: AJaThX5BZHYGHObQZDtfyLAUpyYVsVLZutzdHZbo309NyyLuRT6/8GJ5 fkSfZJAiTklRaeILEAEIgcJ5q0zz88DcfBdpLZH0cRG2id8=
X-Google-Smtp-Source: AGs4zMY9pDSz+gmee+M9EIEuckM0bBWErvKyK6JDEQoFrAO7NZWU1iSMn+21CxTYe0fKo78zfly7VVpJxEk73iIlnhU=
X-Received: by 10.129.87.210 with SMTP id l201mr14580006ywb.2.1512510873467; Tue, 05 Dec 2017 13:54:33 -0800 (PST)
MIME-Version: 1.0
Received: by 10.129.123.132 with HTTP; Tue, 5 Dec 2017 13:53:52 -0800 (PST)
In-Reply-To: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
References: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
From: Eric Rescorla <ekr@rtfm.com>
Date: Tue, 5 Dec 2017 13:53:52 -0800
Message-ID: <CABcZeBPhRP9TNNUjvLZA6vOhz5y40VLF28bdqQYn1iePOewdLg@mail.gmail.com>
To: "tls@ietf.org" <tls@ietf.org>
Content-Type: multipart/alternative; boundary="001a11457576551a95055f9ee1a9"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/or-8GSOSIpEbEQk7tkL7XPyrDHY>
Subject: Re: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Dec 2017 21:54:37 -0000

On Tue, Dec 5, 2017 at 1:35 PM, Eric Rescorla <ekr@rtfm.com>; wrote:

> Hi folks,
>
> I now have some preliminary numbers to share with the group based on
> our Firefox experiments. The executive summary is that our data
> confirms Google's results. More detail below.
>
>
> EXPERIMENTAL DESIGN
> This is a forced experiment in which each client tries all the
> variants. The experiment is deployed via a system add-on (a remotely
> deployable, centrally managed piece of JavaScript code), and then
> takes measurements by trying to do an XHR to a given URL
> (https://mail.google.com/robots.txt) with a specific set of flags. We
> do the following three measurements:
>
> - TLS 1.2
> - TLS 1.3 draft-18
> - TLS 1.3 draft-18 with (approximately) PR#1092 ("7e02")
>
> We take five trials for each measurement, randomly shuffling the
> measurement order and then repeating the shuffled pattern five
> times. Each trial is done with a different connection and we declare
> "success" when any of the five trials succeeds.
>
>
> RESULTS
> This experiment was run on a 2% sample of the Firefox Beta population
> who have locale set to en-US, which we selected because of very
> high GMail blocking rates in some locales, which is a potential
> confounding factor. The experimen started 11/27 and has been running
> through today.
>
> This gave us an initial population of 161578, of whom 160809 (99.5%
> completed the experiment and reported results). This produced the
> following results:
>
>                      Success      Failure      Fail Rate
> --------------------------------------------------------
>
> TLS 1.2               158260         2549          .0158
> TLS 1.3-18            158194         4743          .0291
>

Oops. This first number should be 156066. This is what happens when you cut
and paste from your notebook.

-Ekr


TLS 1.3-Experiment    158194         2615          .0163
>
> For the statistics minded, the difference between -18 and 1.2 is
> significant at p < .001 and the 95% confidence interval of the failure
> rate difference is .0122-.0143 (using R's prop.test). There is no
> significant difference between 1.2 and 1.3-experiment (p = .36).
>
> We've got a -22 experiment in flight now, but it will only be on
> Nightly, so this is probably the strongest data we will have for
> a while.
>
> -Ekr
>
>
> ADDITIONAL DETAILS
> The relevant NSS version: https://dxr.mozilla.org/
> mozilla-beta/source/security/nss/lib/ssl
> Experimental code: https://github.com/mozilla/one-off-system-add-ons/tree/
> master/addons/tls13-middlebox-ghack
> iPython Notebook with analysis: https://gist.github.com/ekr/
> 598208b5399faf303453b10cb11647bf
>
>
>
>
>
>
>