Re: [TLS] More compatibility measurement results

Sean Turner <sean@sn3rd.com> Sat, 23 December 2017 14:30 UTC

Return-Path: <sean@sn3rd.com>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5DC0C1270B4 for <tls@ietfa.amsl.com>; Sat, 23 Dec 2017 06:30:49 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.698
X-Spam-Level:
X-Spam-Status: No, score=-2.698 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=sn3rd.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MeQI9xITsIiy for <tls@ietfa.amsl.com>; Sat, 23 Dec 2017 06:30:47 -0800 (PST)
Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D8155126B72 for <tls@ietf.org>; Sat, 23 Dec 2017 06:30:46 -0800 (PST)
Received: by mail-oi0-x231.google.com with SMTP id w131so20617353oiw.0 for <tls@ietf.org>; Sat, 23 Dec 2017 06:30:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sn3rd.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=B+YdKx/dEbJsFVAXXr895P/5YSh2fqJb+YwwZt05EP8=; b=hSI2GODWw3/Z/3bemHQInAeGEfu/DpZsijbvRiMpYMk1CgawdoT9NJpLpIYBBTihy+ OUSAjWYQU742QftPVl5skDfXBF2fiNVmU9AeMLgsB05nao33tU09Zxtec7PajL8MfKug N6k4j/NM4fzc7tx+IlpgStRP7viJi51h8ZVfY=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=B+YdKx/dEbJsFVAXXr895P/5YSh2fqJb+YwwZt05EP8=; b=XMDr6zAuR+Ft9WyTH1n+t5CK6uC8giAs2F1qJ/VrM8bt8ftl+6h1dZ7fakxppTwYNl B8yf2rzVsZEbCQIq8+ezQk4Ut8iwKfovYuRcIQXRyj4CyYxJBjGesLe/iINGuvMvbhW4 bcZN/5UZjBr9yDgTtxNO9Uhc4a4J+k4dss+qE9WxPaQyfg5cApaJE9emQDSCVA+uMHf6 9r9XUaDwr+249kxD/8gpDKklPbUSDKLPDfOdfe1VzrTjI9MCTrrc10MKI3rPFUNv3PHd wuFYUoOyLb7oxg9s3425SgGb3eI5N0o8mnSaale8/DAUCAJIAXZdL98ZqBwh5WW2+upZ b8KQ==
X-Gm-Message-State: AKGB3mKXe9qHGyp2wybY3gkJhqdGZkncyeYugupkek9GShTh/ZO/EnPn L2m3HaD0Ho7nmFwHreDyY7NEZIgk1H4=
X-Google-Smtp-Source: ACJfBouas6pctqQHVTAzxazuKMRtD0Cb9jbYFwP7CDpz+vfzZzzAlLHLkM0Iud+YGBh4NKw4Eh4YFw==
X-Received: by 10.202.1.201 with SMTP id 192mr13089159oib.97.1514039446006; Sat, 23 Dec 2017 06:30:46 -0800 (PST)
Received: from [10.83.5.240] (mobile-166-173-58-124.mycingular.net. [166.173.58.124]) by smtp.gmail.com with ESMTPSA id d55sm1577756otf.2.2017.12.23.06.30.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 23 Dec 2017 06:30:45 -0800 (PST)
Content-Type: multipart/alternative; boundary=Apple-Mail-294A8685-345B-4A54-B4EC-14CDFE87AAF2
Mime-Version: 1.0 (1.0)
From: Sean Turner <sean@sn3rd.com>
X-Mailer: iPhone Mail (15C153)
In-Reply-To: <CABcZeBMKAYFzA+a87GW_z=oJCqNqCsbhffHswa9dyCRJz5u5+A@mail.gmail.com>
Date: Sat, 23 Dec 2017 08:30:43 -0600
Cc: tls@ietf.org
Content-Transfer-Encoding: 7bit
Message-Id: <9480B42E-8C17-4282-93D1-9763BEAD2453@sn3rd.com>
References: <CABcZeBMKAYFzA+a87GW_z=oJCqNqCsbhffHswa9dyCRJz5u5+A@mail.gmail.com>
To: Eric Rescorla <ekr@rtfm.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/KiTdlVC8K5oW94fWV2xmP5Ln62g>
Subject: Re: [TLS] More compatibility measurement results
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Dec 2017 14:30:49 -0000

Thanks for these!

spt

Sent from my iPhone

> On Dec 22, 2017, at 14:00, Eric Rescorla <ekr@rtfm.com> wrote:
> 
> Hi folks,
> 
> Here are the results of our experiment with Firefox Nightly (draft-22)
> against Facebook.
> 
> EXPERIMENTAL DESIGN
> This is a forced experiment in which each client tries all the
> variants. The experiment is deployed via a system add-on (a remotely
> deployable, centrally managed piece of JavaScript code), and then
> takes measurements by trying to do an XHR to a given URL
> (https://www.tls13.facebook.com/) with a specific set of flags. We
> do the following three measurements:
> 
> - TLS 1.2
> - TLS 1.3 draft-22
> - TLS 1.3 draft-22 in compat mode.
> 
> We take five trials for each measurement, randomly shuffling the
> measurement order and then repeating the shuffled pattern five
> times. Each trial is done with a different connection and we declare
> "success" when any of the five trials succeeds.
> 
> 
> RESULTS
> This experiment was run on a 40% sample of the Firefox Nightly population
> who have locale set to en-US. The data below is taken from the period
> 20171216 to 20171222. There's a bit of contamination in the targeting
> because we temporarily failed to filter on "en-US", but that should
> mostly only affect the first day.
> 
> 37716 clients started the experiment and 37430 completed it (99.2%).
> 
> The results are:
> 
>                                     Success         Fail         Rate
>                       fb-tls12        35615         1815     0.048491
>              fb-tls13-draft-22        35552         1878     0.050174
>        fb-tls13-draft22-compat        35630         1800     0.048090
> 
> The overall failure numbers here are a lot higher than with our Beta
> experiment, which may be a result of different targeting on Nightly
> versus Beta. In particular, I'm still seeing a lot of data from China
> and Vietnam, which seem to have high blocking rates in general (i.e.,
> not just for TLS 1.3). If I restrict to non China and non-Vietnam, we
> get:
> 
>                                     Success         Fail         Rate
>                       fb-tls12        35034         1176     0.032477
>              fb-tls13-draft-22        34960         1250     0.034521
>        fb-tls13-draft22-compat        35037         1173     0.032394
> 
> None of these differences are statistically significant (in the second
> data set, the p value for 1.2 versus -22 is .13), but this all seems
> consistent with saying that that -22 compat mode isn't significantly
> worse than TLS 1.2 and that normal -22 may be somewhat worse
> (unfortunately, we don't have -18 in this experiment).
> 
> Taken together with the results David has reported and our previously
> reported Beta results, this seems fairly encouraging. We'll probably
> let the Nightly experiment run a little longer to see if we hit significance,
> but after that will start looking at a rollout of -22 to Release.
> 
> -Ekr
> 
> 
> ADDITIONAL DETAILS
> Experimental code: https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-draft22
> 
> _______________________________________________
> TLS mailing list
> TLS@ietf.org
> https://www.ietf.org/mailman/listinfo/tls