Re: H2 Server Push performance data

Tom Bergan <> Mon, 22 July 2019 18:06 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id D93BC1200E3 for <>; Mon, 22 Jul 2019 11:06:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.751
X-Spam-Status: No, score=-2.751 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 4IJKHx_Zd3ya for <>; Mon, 22 Jul 2019 11:06:22 -0700 (PDT)
Received: from ( [IPv6:2603:400a:ffff:804:801e:34:0:38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 49531120089 for <>; Mon, 22 Jul 2019 11:06:22 -0700 (PDT)
Received: from lists by with local (Exim 4.89) (envelope-from <>) id 1hpcez-0008Rl-TE for; Mon, 22 Jul 2019 18:03:41 +0000
Resent-Date: Mon, 22 Jul 2019 18:03:41 +0000
Resent-Message-Id: <>
Received: from ([2603:400a:ffff:804:801e:34:0:4f]) by with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <>) id 1hpcex-0008Qz-B4 for; Mon, 22 Jul 2019 18:03:39 +0000
Received: from ([2607:f8b0:4864:20::b2c]) by with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from <>) id 1hpceP-00042h-Su for; Mon, 22 Jul 2019 18:03:31 +0000
Received: by with SMTP id x188so10645847yba.8 for <>; Mon, 22 Jul 2019 11:02:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qHX3JtfSC5AW19PLekxmIeg4O79XN3VtIDKOXIMqO5A=; b=bCRmNakplFDgGs90p3B6z7BJNW02+Ehtk906dAtM6JqcADiVs/Ud+JTH4RkqGQPAco 8yT33pfcrNGctWOuS3H3hb37T1fuye9OzIgHz8TW5OqMYFQI4YNPA/m12qT4bCBidKT1 bXGAuzfsvUc1UVGqiZD+PyiopXyHPi/N1GvAU=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qHX3JtfSC5AW19PLekxmIeg4O79XN3VtIDKOXIMqO5A=; b=AOFGhqSzcV8onlHRQ51/YZmZL9Fm10VEnVIXf4hKpWHJf4zBo94WZ7DYToFUwaep8/ eqH5LWmrTc61OjYo3lqllPw+KCudfCXizrMZFeNvm04+Bp2LlfhVIRSuxtqweJslffqd Hs2uvkwhfWsfqImxYjrVEGpRTAPda4HO/Wu0MpGGDOvCbzvbznwdWJHkcW/l3yMKXAJ6 gzM5Wzw84YW+jdANrd+8kPaZX4WSmCzMkxuWX008xizQkMzMqKVsq/Wubk9qQeoRMip7 pvVhTpOd3J/Vcw4H7wPLEYB9SABjTj/gxu6sih59WDcDyLgWs9nx5utoXFboCTh0sc5S J5ig==
X-Gm-Message-State: APjAAAVslTBRPvd/b8OxTBRfiLtUdS2HszdOhhCIeuDZNMwbKNQSJRV7 jM1spenLGL5VQ9foBsYeHane40tLmFU=
X-Google-Smtp-Source: APXvYqwwskBNgFH8isTk/GnwFyb0qN0Tkdgf2GsfG8O5cq1z1YUAzqU7k/dXAzTHavRMgbJoqes7ew==
X-Received: by 2002:a25:d10a:: with SMTP id i10mr46551926ybg.101.1563818562846; Mon, 22 Jul 2019 11:02:42 -0700 (PDT)
Received: from ( []) by with ESMTPSA id z6sm9845869ywg.40.2019. for <> (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Mon, 22 Jul 2019 11:02:42 -0700 (PDT)
Received: by with SMTP id x74so16116373ywx.6 for <>; Mon, 22 Jul 2019 11:02:42 -0700 (PDT)
X-Received: by 2002:a0d:ea06:: with SMTP id t6mr38991166ywe.186.1563818561753; Mon, 22 Jul 2019 11:02:41 -0700 (PDT)
MIME-Version: 1.0
References: <>
In-Reply-To: <>
From: Tom Bergan <>
Date: Mon, 22 Jul 2019 11:02:29 -0700
X-Gmail-Original-Message-ID: <>
Message-ID: <>
To: "Nanner, Aman" <>
Cc: "" <>
Content-Type: multipart/alternative; boundary="000000000000de571d058e48e124"
Received-SPF: pass client-ip=2607:f8b0:4864:20::b2c;;
X-W3C-Hub-Spam-Status: No, score=-3.9
X-W3C-Hub-Spam-Report: AWL=0.201, BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: 1hpceP-00042h-Su b1855ded493ce055deb830a6868ea078
Subject: Re: H2 Server Push performance data
Archived-At: <>
X-Mailing-List: <> archive/latest/36816
Precedence: list
List-Id: <>
List-Help: <>
List-Post: <>
List-Unsubscribe: <>

Hi Aman,

Interesting results, thanks for sharing! I do wonder why you decided to
compare means instead of percentiles? It looks like there are some crazy
bad outliers. Assuming your "April 2019 Results with Old Methodology" and
"April 2019 Results (1% quantile excluded)" graphs use the same set of
sites in the same order, the mean goes from about +500ms to -200ms. If 99%
of requests on that site take -200ms, the last 1% must take 70s! Instead of
means, you could have compared medians, or 25th/75th percentiles, or other
percentiles in the distribution. You can compute confidence intervals on
percentiles. You can also do something like Mann-Whitney to check if the
distributions differ significantly.

I'd also like to hear more about this linear regression that you run on the
A/B results. I couldn't follow why this regression is necessary. If your
A/B test uses i.i.d. coin flips, you can compute confidence intervals
directly on the A/B results. Is your A/B test not i.i.d.? How do you know
the linear regression doesn't introduce bias?

Also, have you looked into measuring other metrics, such as


On Mon, Jul 22, 2019 at 4:41 AM Nanner, Aman <> wrote:

> At the previous IETF 102 HTTPWG session in Montreal, I had presented some
> Akamai data on H2 Server Push which can be found here:
> Akamai has conducted some more recent tests with a tweaked methodology
> (exclusion of 1% highest-latency requests on the long-tail), and we have
> found some interesting results. I share some more details about the
> performance analysis here:
> Thanks,
> Aman Nanner
> Akamai Technologies Inc.