Re: [TLS] More compatibility measurement results

Ilari Liusvaara <ilariliusvaara@welho.com> Sat, 23 December 2017 14:07 UTC

Return-Path: <ilariliusvaara@welho.com>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ABB6712D77C for <tls@ietfa.amsl.com>; Sat, 23 Dec 2017 06:07:22 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.91
X-Spam-Level:
X-Spam-Status: No, score=-1.91 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, T_RP_MATCHES_RCVD=-0.01] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hX-ejQUxj4az for <tls@ietfa.amsl.com>; Sat, 23 Dec 2017 06:07:20 -0800 (PST)
Received: from welho-filter3.welho.com (welho-filter3.welho.com [83.102.41.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5281312D779 for <tls@ietf.org>; Sat, 23 Dec 2017 06:07:19 -0800 (PST)
Received: from localhost (localhost [127.0.0.1]) by welho-filter3.welho.com (Postfix) with ESMTP id AB0295DE6D; Sat, 23 Dec 2017 16:07:17 +0200 (EET)
X-Virus-Scanned: Debian amavisd-new at pp.htv.fi
Received: from welho-smtp3.welho.com ([IPv6:::ffff:83.102.41.86]) by localhost (welho-filter3.welho.com [::ffff:83.102.41.25]) (amavisd-new, port 10024) with ESMTP id KwFxQWWthOPa; Sat, 23 Dec 2017 16:07:17 +0200 (EET)
Received: from LK-Perkele-VII (87-92-19-27.bb.dnainternet.fi [87.92.19.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by welho-smtp3.welho.com (Postfix) with ESMTPSA id 6B0112308; Sat, 23 Dec 2017 16:07:15 +0200 (EET)
Date: Sat, 23 Dec 2017 16:07:15 +0200
From: Ilari Liusvaara <ilariliusvaara@welho.com>
To: Eric Rescorla <ekr@rtfm.com>
Cc: tls@ietf.org
Message-ID: <20171223140714.GA29043@LK-Perkele-VII>
References: <CABcZeBMKAYFzA+a87GW_z=oJCqNqCsbhffHswa9dyCRJz5u5+A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
In-Reply-To: <CABcZeBMKAYFzA+a87GW_z=oJCqNqCsbhffHswa9dyCRJz5u5+A@mail.gmail.com>
User-Agent: Mutt/1.9.2 (2017-12-15)
Sender: ilariliusvaara@welho.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/ebFKQP_4TBrjRq2bBPFsowL4e8s>
Subject: Re: [TLS] More compatibility measurement results
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Dec 2017 14:07:23 -0000

On Fri, Dec 22, 2017 at 12:00:12PM -0800, Eric Rescorla wrote:
> Hi folks,
> 
> Here are the results of our experiment with Firefox Nightly (draft-22)
> against Facebook.
> 
> 
> RESULTS
> 
> 37716 clients started the experiment and 37430 completed it (99.2%).
> 
> The results are:
> 
>                                     Success         Fail         Rate
>                       fb-tls12        35034         1176     0.032477
>              fb-tls13-draft-22        34960         1250     0.034521
>        fb-tls13-draft22-compat        35037         1173     0.032394
> 
> None of these differences are statistically significant (in the second
> data set, the p value for 1.2 versus -22 is .13), but this all seems
> consistent with saying that that -22 compat mode isn't significantly
> worse than TLS 1.2 and that normal -22 may be somewhat worse
> (unfortunately, we don't have -18 in this experiment).
>
> Taken together with the results David has reported and our previously
> reported Beta results, this seems fairly encouraging. We'll probably
> let the Nightly experiment run a little longer to see if we hit
> significance,
> but after that will start looking at a rollout of -22 to Release.

~3.25% baseline failure rate? That sounds quite high. ~0.2% above-
baseline failure rate for non-compat? That sound fairly low, but
there have been improvements here that could have caused substantial
decrease.

I wonder if the high baseline failure rate is due to high amount of
blocking of the test server. And unfortunately, the places that
blocked the test server are some of the most interesting when it comes
to the compatibility.

However, the results do establish that the incremential failure rates
in open environments (anything that blocks the testserver very probably
is not open environment) are low enough to proceed with.


-Ilari