[TLS] More compatibility measurement results

Eric Rescorla <ekr@rtfm.com> Fri, 22 December 2017 20:01 UTC

Return-Path: <ekr@rtfm.com>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 07D3B127735 for <tls@ietfa.amsl.com>; Fri, 22 Dec 2017 12:01:21 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=rtfm-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id hwE0d8rhbteS for <tls@ietfa.amsl.com>; Fri, 22 Dec 2017 12:01:18 -0800 (PST)
Received: from mail-yb0-x234.google.com (mail-yb0-x234.google.com [IPv6:2607:f8b0:4002:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 76D1812741D for <tls@ietf.org>; Fri, 22 Dec 2017 12:00:54 -0800 (PST)
Received: by mail-yb0-x234.google.com with SMTP id q3so6521563ybg.9 for <tls@ietf.org>; Fri, 22 Dec 2017 12:00:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rtfm-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=+Xiyh/SdFgAv6rgWaOk5GY6q0pgeOgcdSeB7qSzQc90=; b=W7BOLG3P4YdbGvBEUzSLzlYxurnHwnPdwXTIMVPUpgGCdt7Jbvl4IDhbq4tt40hluo McDzpbkMfdiSk8cC809cslVlQy+dSRiBBk8oHNqBJD8iJe+DkIHt4lfKWMLVtuaFdcw1 lUjVorg0QTGHzQWzGcG616T8wqGhOGdV6rZGLyMULd//YSgTI7sF7l/opNodOZIwyH2b 8MRF9oYQ2Y0vKh+o9mOg+Kb1ufnEjN2FyNAF39z9xylvygEzGEEytdD6nJrVn8zKRnry l9oXejU7emdEZsbwDZSE6nHpKQcQTE72vTd7nmuGr+tvKFz1rSdOGvyZkCKLdNHaLuu7 A9fA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=+Xiyh/SdFgAv6rgWaOk5GY6q0pgeOgcdSeB7qSzQc90=; b=OILjfi8lQGxFPXMGhqmuQgOErNJrIrwul1ab7WaTtAAPiL1ZhGhra8o4hIhZi8b0/c g/z/tVPrO+MJU2GkcD5hVuT/qP/KVYhH9MU4vYG84dSiV2yyqCbuF+WKSn2RIz7WKLsG 9rqNNFhIgby7+GdgQ0C/N4mLBUz3ft832TvZHO8btkPNatkc/0qCEl/kSxpuFeAdlo8g 5wzsx2i9AUoZaakHIyY0gsf9hOihHjq7PsusypfcYoIz8/kruZKAhc3dzkaDaUJmHrWX ZbftoU62sqQ6NeX6eeqhEcGqNDtFQPgFLuJFF6I2vcO5PmhgvRcnOkcwlUfI+c57DpPD egQA==
X-Gm-Message-State: AKGB3mI6ACu9hkyHhTQ41thEaMZeieI58uO1EABO6QOXZuG/Jz5v7mGD 4QtPpla3zDwn/Wmfu9tAzQrBaam1uIr5xly1D2yW6yzTOKA=
X-Google-Smtp-Source: ACJfBovTYTL371Zml6GX6usN3e598dNTirtHOwgBFwhyb3NHkT7N0WGmUHaHlCkfsDhDGknCdGVywTEvAJbIWGXpLPw=
X-Received: by with SMTP id x4mr11455900ybg.200.1513972853337; Fri, 22 Dec 2017 12:00:53 -0800 (PST)
MIME-Version: 1.0
Received: by with HTTP; Fri, 22 Dec 2017 12:00:12 -0800 (PST)
From: Eric Rescorla <ekr@rtfm.com>
Date: Fri, 22 Dec 2017 12:00:12 -0800
Message-ID: <CABcZeBMKAYFzA+a87GW_z=oJCqNqCsbhffHswa9dyCRJz5u5+A@mail.gmail.com>
To: tls@ietf.org
Content-Type: multipart/alternative; boundary="94eb2c084a941f89e20560f3461e"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/6pGGT-wm5vSkacMFPEPvFMEnj-M>
Subject: [TLS] More compatibility measurement results
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Dec 2017 20:01:21 -0000

Hi folks,

Here are the results of our experiment with Firefox Nightly (draft-22)
against Facebook.

This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://www.tls13.facebook.com/) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-22
- TLS 1.3 draft-22 in compat mode.

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.

This experiment was run on a 40% sample of the Firefox Nightly population
who have locale set to en-US. The data below is taken from the period
20171216 to 20171222. There's a bit of contamination in the targeting
because we temporarily failed to filter on "en-US", but that should
mostly only affect the first day.

37716 clients started the experiment and 37430 completed it (99.2%).

The results are:

                                    Success         Fail         Rate
                      fb-tls12        35615         1815     0.048491
             fb-tls13-draft-22        35552         1878     0.050174
       fb-tls13-draft22-compat        35630         1800     0.048090

The overall failure numbers here are a lot higher than with our Beta
experiment, which may be a result of different targeting on Nightly
versus Beta. In particular, I'm still seeing a lot of data from China
and Vietnam, which seem to have high blocking rates in general (i.e.,
not just for TLS 1.3). If I restrict to non China and non-Vietnam, we

                                    Success         Fail         Rate
                      fb-tls12        35034         1176     0.032477
             fb-tls13-draft-22        34960         1250     0.034521
       fb-tls13-draft22-compat        35037         1173     0.032394

None of these differences are statistically significant (in the second
data set, the p value for 1.2 versus -22 is .13), but this all seems
consistent with saying that that -22 compat mode isn't significantly
worse than TLS 1.2 and that normal -22 may be somewhat worse
(unfortunately, we don't have -18 in this experiment).

Taken together with the results David has reported and our previously
reported Beta results, this seems fairly encouraging. We'll probably
let the Nightly experiment run a little longer to see if we hit
but after that will start looking at a rollout of -22 to Release.


Experimental code: