Re: Experiences with HTTP/2 server push

Bryan McQuade <bmcquade@google.com> Sun, 04 December 2016 12:49 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 231A51294E4 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sun, 4 Dec 2016 04:49:51 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.396
X-Spam-Level:
X-Spam-Status: No, score=-9.396 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_SORBS_SPAM=0.5, RP_MATCHES_RCVD=-2.896, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NtsJud10-Cak for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Sun, 4 Dec 2016 04:49:48 -0800 (PST)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1F9141294E0 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Sun, 4 Dec 2016 04:49:47 -0800 (PST)
Received: from lists by frink.w3.org with local (Exim 4.80) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1cDWBX-000281-DK for ietf-http-wg-dist@listhub.w3.org; Sun, 04 Dec 2016 12:46:27 +0000
Resent-Date: Sun, 04 Dec 2016 12:46:27 +0000
Resent-Message-Id: <E1cDWBX-000281-DK@frink.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by frink.w3.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from <bmcquade@google.com>) id 1cDWBF-00026J-SK for ietf-http-wg@listhub.w3.org; Sun, 04 Dec 2016 12:46:09 +0000
Received: from mail-io0-f170.google.com ([209.85.223.170]) by mimas.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <bmcquade@google.com>) id 1cDWB4-0000rg-P3 for ietf-http-wg@w3.org; Sun, 04 Dec 2016 12:46:04 +0000
Received: by mail-io0-f170.google.com with SMTP id a124so555182937ioe.2 for <ietf-http-wg@w3.org>; Sun, 04 Dec 2016 04:45:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Rvelt6+htko1or32ME+6+vk6GPijKUf90GrzYUrV5Hw=; b=FHSCHhlg0mFd63OhbkOv1SOUp4I5JQYHUOSRTuEynaa1o3cWeCJHEwPOy3K5y8Ki71 mi3xNV6AjuQp0w+hfTTZBP1VYx6HaC+tFnQgUX26Hh50sp8GxmmF0PtgagP4p+/ncXzo 51oWBsOoSW3VizheE9pjMzTNr5Zle7yvhQVXHn8CVbVM1Ce1/dOz+x2LvReAXdcK+a+f buFBOPNmuw/uujjOYtKtSZyAKxUa2UGS57OdEhy1Gy/siUy32vgto5n6PwsoffNIYKrs ykDR1CsIcNvlcmghgKr2uVl5wCd+FAilprAwoynnUmPtLsV/gl84e6q+DCdMu/+EUxOZ L8Bg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Rvelt6+htko1or32ME+6+vk6GPijKUf90GrzYUrV5Hw=; b=J5v+jTgzBVS6GmaPvZh1KxNzW2TJMlFfR8poBeu3SukmsPuFqSCoTascVM+EqwDW7C eeEabNHPRWCxJsWVmMPu1WXwVA5J1o7I5eFPwr2ok23hKkpQXFUiMAszYV3+f91dIDYK geQvTuCMbPVuLwzMnhi8Y+1ZgjfWXLXjUZhuaWthcbc7I5reovZMjVmWygZ7iFJteaPu ZgtZmAhW9e9nx/R66EL5m7m7jUGnTfdskFl2c1xumra6vFhg+pzYPg8fJFC0sw6yLg6c yp053DJZHPf8WsvNds3OCiHHfbeM12K42U+cFuVUeOxO5jqwG/iioqQRnAfS2NGTbjqT k+uQ==
X-Gm-Message-State: AKaTC00fWYWc7kyXpEOfSLgMjgQbuZ7SnIAZDlaUeDkN/mxyTQr2ju8GarK6qxaIiJ64I48JF5kpluvtvnYgwJm+
X-Received: by 10.36.181.21 with SMTP id v21mr4661003ite.75.1480855522175; Sun, 04 Dec 2016 04:45:22 -0800 (PST)
MIME-Version: 1.0
References: <CACZw55mUg_VjN3Q6TqPCb6udo3mQpoWQVNV5e2iYiNj=hC-2kw@mail.gmail.com> <CANatvzz0rBjkgV4yS+2hgspUj7wZ6y=NqojPyzHiPpvZVXzwEA@mail.gmail.com> <CA+3+x5F8M+xbH2YjD9m87WPOyQPCTVNV8evBJQHn9gicch+1TQ@mail.gmail.com> <CANatvzwdmz4L06DbzxBgm1+YeVcpYXDck2QBEXoytHfoJBDzZg@mail.gmail.com>
In-Reply-To: <CANatvzwdmz4L06DbzxBgm1+YeVcpYXDck2QBEXoytHfoJBDzZg@mail.gmail.com>
From: Bryan McQuade <bmcquade@google.com>
Date: Sun, 04 Dec 2016 12:45:10 +0000
Message-ID: <CADLGQyAm5yTr+RFEnZ=RhwRg6S42CY=OooRTw8DA9ta_YDiihA@mail.gmail.com>
To: Kazuho Oku <kazuhooku@gmail.com>, Tom Bergan <tombergan@chromium.org>
Cc: HTTP Working Group <ietf-http-wg@w3.org>
Content-Type: multipart/alternative; boundary=f403045d9ab45d20910542d48b58
Received-SPF: pass client-ip=209.85.223.170; envelope-from=bmcquade@google.com; helo=mail-io0-f170.google.com
X-W3C-Hub-Spam-Status: No, score=-6.4
X-W3C-Hub-Spam-Report: AWL=0.031, BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, RP_MATCHES_RCVD=-2.896, SPF_PASS=-0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1cDWB4-0000rg-P3 732dd20c5508e346fa23255bf8fa4c3e
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Experiences with HTTP/2 server push
Archived-At: <http://www.w3.org/mid/CADLGQyAm5yTr+RFEnZ=RhwRg6S42CY=OooRTw8DA9ta_YDiihA@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/33104
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

Here's a new article on H2 push from today's perf calendar which goes into
a good bit of detail:
http://calendar.perfplanet.com/2016/http2-push-the-details/

On Thu, Sep 1, 2016 at 5:39 PM Kazuho Oku <kazuhooku@gmail.com> wrote:

> Hi,
>
> Thank you for your response.
>
> 2016-09-02 1:38 GMT+09:00 Tom Bergan <tombergan@chromium.org>rg>:
> > Thanks for the feedback and link to that workshop talk! A few comments
> > inline.
> >
> > On Wed, Aug 31, 2016 at 9:57 PM, Kazuho Oku <kazuhooku@gmail.com> wrote:
> >>
> >> Consider the case where a large HTML that loads a CSS is sent over the
> >> wire. In a typical implementation, the server will pass a block of
> >> HTML much larger than INITCWND to the TCP stack before recognizing the
> >> request for CSS. So the client would need to wait for multiple RTTs
> >> before starting to receive the CSS.
> >
> >
> > Unrelated to your above comment -- I think servers should use a higher
> > initcwnd with H2, and I know that some servers do this. The experiments
> in
> > our doc used Linux's default initcwnd (10 packets). If you compare that
> to
> > H1, where browsers use 6 concurrent connections, the effective initcwnd
> for
> > H1 is 60 packets (well, not exactly, since the browser only makes one
> > request initially, but as soon as the browser starts making additional
> > requests, cwnd effectively grows much faster than it would with a single
> > connection).
> >
> >>
> >> That said, as discussed at the workshop, it is possible to implement a
> >> HTTP/2 server that does not get affected by HoB between the different
> >> streams (see
> >>
> https://github.com/HTTPWorkshop/workshop2016/blob/master/talks/tcpprog.pdf
> ).
> >>
> >> I would suggest that regardless of whether or not push is used, server
> >> implementors should consider adopting such approach to minimize the
> >> impact of HoB.
> >
> >
> > This is really interesting. To summarize: the idea is to use getsockopt
> to
> > compute the number of available bytes in cwnd so that sizeof(kernel
> buffer)
> > = cwnd. I rejected this idea without thinking about it much because it
> > seemed like it would increase kernel/user round-trips and perform poorly
> in
> > bursty conditions. But, your idea to restrict this optimization to cases
> > where it matters most makes sense. Do you have performance measurements
> of
> > this idea under heavy load?
>
> Unfortunately not.
>
> I agree that it would be interesting to collect metrics based on real
> workload, both on the client side and the server side.
>
> OTOH let me note that since we enable the optimization only for
> connections with RTT substantially higher than the time spent by a
> single iteration of the event loop, we expect that there would be no
> performance penalty when facing a burst. The server would just switch
> to the ordinary way.
>
> > Are you using TCP_NOTSENT_LOWAT for cases where
> > the optimization cannot be used?
>
> No. I'm not sure if restricting the amount of unsent data to a fixed
> value is generally a good thing, or if that causes practical impact on
> performance.
>
> Personally, for connections that left the slow-start phase, I prefer
> the amount calculated proportional to the current CWND value, which
> IIRC is the default behavior of Linux.
>
> >>
> >> It should also be noted that with QUIC such HoB would not be an issue
> >> since there would no longer be a send buffer within the kernel.
> >
> >
> > Yep, this is definitely an advantage of QUIC.
> >
> >> "Rule 2: Push Resources in the Right Order"
> >>
> >> My take is that the issue can / should be solved by clients sending
> >> PRIORITY frames for pushed resources when they observe how the
> >> resources are used, and that until then servers should schedule the
> >> pushed streams separately from the client-driven prioritization tree
> >> (built by using the PRIORITY frames).
> >>
> >> Please refer to the discussion in the other thread for details:
> >> https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0453.html
> >
> >
> > To make sure I understand the idea: Suppose you send HTML, then push
> > resources X and Y. You will continue pushing X and Y until you get
> requests
> > from the client, at which point you switch to serving requests made by
> the
> > client (which may or may not include X and Y, as the client may not know
> > about those resources yet, depending on what you decided to push). These
> > client requests are served via the client-driven priority tree.
> >
> > Is that right? If so, you've basically implemented rule #1
>
> Actually not.
>
> My interpretation of rule #1 (or the solution proposed for rule #1)
> was that it discusses the impact of TCP-level head-of-line blocking,
> whereas rule #2 seemed to discuss the issues caused by pushed streams
> not appropriately prioritized against the pulled streams.
>
> And the solution for rule #2 that I revisited here was for a server to
> prioritize _some_ of the pushed streams outside the client-driven
> priority tree.
>
> I am not copy-pasting the scheme described in
> https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0453.html
> in fear that doing so might lose context, but as an example, it would
> go like this.
>
> Suppose you are sending HTML (in response to a pull), as well as
> pushing two asset files: one is CSS and one is an image.
>
> Among the two assets, it is fair for a server to anticipate that the
> CSS is likely to block the rendering of the HTML. Therefore, the
> server sends CSS before HTML (but does not send a PRIORITY frame for
> the CSS, since PRIORITY frame is a tool for controlling client-driven
> prioritization). OTOH an image is not likely to block the rendering.
> Therefore, it is scheduled as specified by the HTTP/2 specification
> (so that it would be sent after the HTML).
>
> This out-of-client-driven-priotization-tree scheduling should be
> performed until a server receives a PRIORITY frame adjusting the
> precedence of a pushed stream. At this point, a server should
> reprioritize the pushed stream (i.e. CSS) if it considers client's
> knowledge of how the streams should be prioritized superior to what
> the server knows.
>
> > -- the push lasts
> > while the network is idle, then you switch to serving client requests
> > afterwards. It's nice to see that we came to the same high-level
> conclusion
> > :-). But, I like the way you've phrased the problem. Instead of
> computing a
> > priori out how much data you should push, which we suggested, you start
> > pushing an arbitrary number of things, then you'll automatically stop
> > pushing as soon as you get the next client request.
> >
> > One more clarification: what happens when the client loads two pages
> > concurrently and the network is effectively never idle? I assume push
> won't
> > happen in this case?
> >
> > Next, I think you're arguing that push order doesn't matter as long as
> you
> > have a solution for HoB. I don't think this is exactly right.
> Specifically:
> >
> > - Head-of-link blocking (HoB) can happen due to network-level
> bufferbloat.
> > The above solution only applies to kernel-level bufferbloat. You need
> some
> > kind of bandwidth-based pacing to avoid network-level buffer bloat.
>
> That's correct.
>
> OTOH I would like to point out that the issue is irrelevant to push.
>
> A client would issue requests in the order it notices the URLs that it
> should fetch. And it cannot update the priority of the links found in
> LRP headers until it observes how the resource is actually used.
>
> So if preload links included low-priority assets, bufferbloat can (or
> will) cause issues for both pull and push.
>
> > - If you're pushing X and Y, and you know the client will use X before Y,
> > you should push in that order. The opposite order is sub-optimal and can
> > eliminate the benefit of push in some cases, even ignoring HoB.
>
> Agreed.
>
> And my understanding is that both Apache and H2O does this, based on
> the content-type of the pushed response.
>
> Just having two (or three) levels of precedence (send before HTML vs.
> send after HTML vs. send along with HTML) is not as complex as what
> HTTP/2's prioritization tree provides, but I think is sufficient for
> optimizing the time spent until first-render.
>
> What would be the best way to prioritize the blocking assets (i.e. an
> asset that needs to be sent before HTML, e.g. CSS) is what Apache and
> H2O disagree. And my proposal (and what H2O does in that respect) is
> that a server should schedule such pushed streams outside the
> prioritization tree (i.e. my response for rule #2).
>
> >> As a server implementor, I have always dreamt of cancelling a push
> >> after sending a PUSH_PROMISE. In case a resource we want to push
> >> exists on a dedicate cache that cannot be reached synchronously from
> >> the HTTP/2 server, the server needs to send PUSH_PROMISE without the
> >> guarantee that it would be able to push a valid response.
> >>
> >> It would be great if we could have an error code that can be sent
> >> using RST_STREAM to notify the client that it should discard the
> >> PUSH_PROMISE being sent, and issue a request by itself.
> >
> >
> > Yes, +1. I've wanted this feature. It sucks that the client won't reissue
> > the requests if they get a RST_STREAM. (At least, Chrome won't do this, I
> > don't know about other browsers.)
>
>
>
> --
> Kazuho Oku
>
>