Re: H2 Server Push performance data

"Nanner, Aman" <ananner@akamai.com> Mon, 22 July 2019 21:34 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DDF861200B6 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Mon, 22 Jul 2019 14:34:43 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.35
X-Spam-Level:
X-Spam-Status: No, score=-2.35 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (2048-bit key) reason="fail (body has been altered)" header.d=akamai.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EGIHkrhxFglD for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Mon, 22 Jul 2019 14:34:41 -0700 (PDT)
Received: from frink.w3.org (frink.w3.org [IPv6:2603:400a:ffff:804:801e:34:0:38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CD64F12001B for <httpbisa-archive-bis2Juki@lists.ietf.org>; Mon, 22 Jul 2019 14:34:40 -0700 (PDT)
Received: from lists by frink.w3.org with local (Exim 4.89) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1hpfvj-0007e0-At for ietf-http-wg-dist@listhub.w3.org; Mon, 22 Jul 2019 21:33:11 +0000
Resent-Date: Mon, 22 Jul 2019 21:33:11 +0000
Resent-Message-Id: <E1hpfvj-0007e0-At@frink.w3.org>
Received: from uranus.w3.org ([128.30.52.58]) by frink.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <ananner@akamai.com>) id 1hpfvh-0007dA-Dl for ietf-http-wg@listhub.w3.org; Mon, 22 Jul 2019 21:33:09 +0000
Received: from www-data by uranus.w3.org with local (Exim 4.89) (envelope-from <ananner@akamai.com>) id 1hpfvh-0003sa-8T for ietf-http-wg@listhub.w3.org; Mon, 22 Jul 2019 21:33:09 +0000
Received: from mimas.w3.org ([2603:400a:ffff:804:801e:34:0:4f]) by frink.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <ananner@akamai.com>) id 1hpfqv-0001B8-1K for ietf-http-wg@listhub.w3.org; Mon, 22 Jul 2019 21:28:13 +0000
Received: from mx0a-00190b01.pphosted.com ([2620:100:9001:583::1]) by mimas.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <ananner@akamai.com>) id 1hpfqs-0000Su-TQ for ietf-http-wg@w3.org; Mon, 22 Jul 2019 21:28:12 +0000
Received: from pps.filterd (m0050093.ppops.net [127.0.0.1]) by m0050093.ppops.net-00190b01. (8.16.0.42/8.16.0.42) with SMTP id x6MLN468004695; Mon, 22 Jul 2019 22:27:48 +0100
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=akamai.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=jan2016.eng; bh=G9PV7UcT/bc3oFrsDK8xIPIz3EyGqz7LVPUpLmU+HvE=; b=SBFp2xgogZSMgvmP0GInFRff2sEMYRmOEaJ7QzmQX3s1/iTwnY9A2/TpaOrWnslU311g UMcJaQquYIdl5ZTrn0COB7UuNKrukcEEEdaV0and4FrjgUC5DBuytDJEvlxDEjipgexC xpyMd/XRrJfYb3fcmF5gOVawRgV46qBRXYoJfxVNyStUSxWowkeZlZgbM1o2LaCoNbI4 GkD9C86ns8zoy6fnaCJXkt9emH1tRnGLOraYHkn+q4hekv9oWwahZ05HJfMJPNiOHidA j4RJ960Bmy1cTntwpZ0EPxV4XfsLt6dTY4diE0BezrLSarzZaIxE+c6xpsEZQ4kjljrP lg==
Received: from prod-mail-ppoint7 (prod-mail-ppoint7.akamai.com [96.6.114.121] (may be forged)) by m0050093.ppops.net-00190b01. with ESMTP id 2twkv0088p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 22 Jul 2019 22:27:48 +0100
Received: from pps.filterd (prod-mail-ppoint7.akamai.com [127.0.0.1]) by prod-mail-ppoint7.akamai.com (8.16.0.27/8.16.0.27) with SMTP id x6MLKJme008558; Mon, 22 Jul 2019 17:27:47 -0400
Received: from email.msg.corp.akamai.com ([172.27.123.31]) by prod-mail-ppoint7.akamai.com with ESMTP id 2twmvr008w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 22 Jul 2019 17:27:47 -0400
Received: from USMA1EX-DAG1MB5.msg.corp.akamai.com (172.27.123.105) by usma1ex-dag1mb2.msg.corp.akamai.com (172.27.123.102) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 22 Jul 2019 17:27:45 -0400
Received: from USMA1EX-DAG1MB5.msg.corp.akamai.com ([172.27.123.105]) by usma1ex-dag1mb5.msg.corp.akamai.com ([172.27.123.105]) with mapi id 15.00.1473.005; Mon, 22 Jul 2019 17:27:45 -0400
From: "Nanner, Aman" <ananner@akamai.com>
To: Tom Bergan <tombergan@chromium.org>
CC: "ietf-http-wg@w3.org" <ietf-http-wg@w3.org>
Thread-Topic: H2 Server Push performance data
Thread-Index: AQHVQC8mCst/N2AULUeeehkGrwysIabXMfCA///15pQ=
Date: Mon, 22 Jul 2019 21:27:45 +0000
Message-ID: <1563830865138.49957@akamai.com>
References: <8EA56456-0299-4D60-B97A-E6FCD27D9231@akamai.com>, <CA+3+x5H9eF5mNGbLU_M3ZN=Jfe4PdstoeeZ-56G1UOZfRdOO4w@mail.gmail.com>
In-Reply-To: <CA+3+x5H9eF5mNGbLU_M3ZN=Jfe4PdstoeeZ-56G1UOZfRdOO4w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [172.28.82.156]
Content-Type: multipart/alternative; boundary="_000_156383086513849957akamaicom_"
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-07-22_15:, , signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907220233
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-07-22_15:2019-07-22,2019-07-22 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 impostorscore=0 clxscore=1011 spamscore=0 priorityscore=1501 lowpriorityscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1904300001 definitions=main-1907220234
Received-SPF: pass client-ip=2620:100:9001:583::1; envelope-from=ananner@akamai.com; helo=mx0a-00190b01.pphosted.com
X-W3C-Hub-Spam-Status: No, score=-4.7
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1hpfqs-0000Su-TQ 6e8de57c8e981f7f921e848449059a96
X-caa-id: 5195e13a1a
X-Original-To: ietf-http-wg@w3.org
Subject: Re: H2 Server Push performance data
Archived-At: <https://www.w3.org/mid/1563830865138.49957@akamai.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/36820
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

Hi Tom,

Thanks for your feedback!

The first couple of charts (based on the multiple customer sites) use the mean as a way to provide a summary view of performance for each individual site.  The last chart in the article is a percentile plot for one customer site which demonstrates a more granular view, but we do not have percentile plots put together for each customer site. It should be possible to do that in the event we were to perform deeper analysis on more sites.

The A/B data was not collected in a way that we would consider to be pure enough to be considered i.i.d coin flips. There exist dimensions where the values can make it more or less likely that Server Push may be applied on specific responses within the production network (e.g. geo, time of day, user-agent), and so the approach was chosen to use linear regression to attempt to control for these variables.

I agree that the FirstContentfulPaint metric would be useful to analyze; we had some limitations in our dataset regarding collection of that metric that resulted in it not being reliable enough to use (it wasn't as widely collected as DOMComplete). There have been some encouraging results seen for this metric based on some preliminary analysis, but we wouldn't consider the data and analysis reliable enough yet for sharing.

Thanks,
Aman


________________________________
From: Tom Bergan <tombergan@chromium.org>
Sent: Monday, July 22, 2019 2:02 PM
To: Nanner, Aman
Cc: ietf-http-wg@w3.org
Subject: Re: H2 Server Push performance data

Hi Aman,

Interesting results, thanks for sharing! I do wonder why you decided to compare means instead of percentiles? It looks like there are some crazy bad outliers. Assuming your "April 2019 Results with Old Methodology" and "April 2019 Results (1% quantile excluded)" graphs use the same set of sites in the same order, the mean goes from about +500ms to -200ms. If 99% of requests on that site take -200ms, the last 1% must take 70s! Instead of means, you could have compared medians, or 25th/75th percentiles, or other percentiles in the distribution. You can compute confidence intervals on percentiles.. You can also do something like Mann-Whitney to check if the distributions differ significantly.

I'd also like to hear more about this linear regression that you run on the A/B results. I couldn't follow why this regression is necessary. If your A/B test uses i.i.d. coin flips, you can compute confidence intervals directly on the A/B results. Is your A/B test not i.i.d.? How do you know the linear regression doesn't introduce bias?

Also, have you looked into measuring other metrics, such as FirstContentfulPaint?

-Tom

On Mon, Jul 22, 2019 at 4:41 AM Nanner, Aman <ananner@akamai.com<mailto:ananner@akamai.com>> wrote:
At the previous IETF 102 HTTPWG session in Montreal, I had presented some Akamai data on H2 Server Push which can be found here:

https://github.com/httpwg/wg-materials/blob/gh-pages/ietf102/akamai-server-push.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_httpwg_wg-2Dmaterials_blob_gh-2Dpages_ietf102_akamai-2Dserver-2Dpush.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=HVAbQ4bnkmcYKMR6_H0nEMfpVIpY1jgJWVBhC_es5Nw&m=JhnrDdJ2FDK0-S_OawS0GoKyXIVn4C8H_c03AveZDRo&s=UzEaCDDavuWPtu3zi_hdJa51YwKMwdzescfn9rpYnlQ&e=>

Akamai has conducted some more recent tests with a tweaked methodology (exclusion of 1% highest-latency requests on the long-tail), and we have found some interesting results. I share some more details about the performance analysis here:

https://medium.com/@ananner/http-2-server-push-performance-a-further-akamai-case-study-7a17573a3317<https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40ananner_http-2D2-2Dserver-2Dpush-2Dperformance-2Da-2Dfurther-2Dakamai-2Dcase-2Dstudy-2D7a17573a3317&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=HVAbQ4bnkmcYKMR6_H0nEMfpVIpY1jgJWVBhC_es5Nw&m=JhnrDdJ2FDK0-S_OawS0GoKyXIVn4C8H_c03AveZDRo&s=5ScQ7t8fvPbCcHySFH1WDh9iHpnjchASXvAmdo8mvJM&e=>

Thanks,
Aman Nanner
Akamai Technologies Inc.