Re: Priority implementation complexity (was: Re: Extensible Priorities and Reprioritization)

Yoav Weiss <yoav@yoav.ws> Mon, 15 June 2020 11:47 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1B9A13A0D05 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Mon, 15 Jun 2020 04:47:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.648
X-Spam-Level:
X-Spam-Status: No, score=-2.648 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=yoav-ws.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6opJwBdori2q for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Mon, 15 Jun 2020 04:47:00 -0700 (PDT)
Received: from lyra.w3.org (lyra.w3.org [128.30.52.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6D1B13A0D01 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Mon, 15 Jun 2020 04:47:00 -0700 (PDT)
Received: from lists by lyra.w3.org with local (Exim 4.92) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1jknXH-0003hg-KO for ietf-http-wg-dist@listhub.w3.org; Mon, 15 Jun 2020 11:44:19 +0000
Resent-Date: Mon, 15 Jun 2020 11:44:19 +0000
Resent-Message-Id: <E1jknXH-0003hg-KO@lyra.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by lyra.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <yoav@yoav.ws>) id 1jknXG-0003gv-2d for ietf-http-wg@listhub.w3.org; Mon, 15 Jun 2020 11:44:18 +0000
Received: from mail-lf1-x12d.google.com ([2a00:1450:4864:20::12d]) by mimas.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from <yoav@yoav.ws>) id 1jknXD-0000bG-9f for ietf-http-wg@w3.org; Mon, 15 Jun 2020 11:44:17 +0000
Received: by mail-lf1-x12d.google.com with SMTP id t74so1748285lff.2 for <ietf-http-wg@w3.org>; Mon, 15 Jun 2020 04:44:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yoav-ws.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3aw1zzPFttGiixqkPWsGHzDixzvsuzy+40Vz1/ujh98=; b=jMcNz5xxSLDCXp8t27re+uQVgviOG4dZ2dq5Rc80f86K56iRFEmE/yA7bq0ncAEJTy JuqQfkWFopqio+w5Qz+CGEJ5nA3S+RVP37P0HdNryLsmrLRUYgI3dah7YWKm6t1EMyrh QnK4KeIi26wy5GIQnXsb6WXG+OXToOdcK0dMESetN/pvvH8+5Z7epVO+TGfgJkl38flu ZJb/81JAkJzSeM8SRF/dlHAFs+2sVptIN9NGS52fZKuaPwIXxvu2iXH3rpmW/2+R6zlO uGzxbLqrczDQipiZBE/lTlhqsORBs9pH/GxHFGS5ZBdKsz5HrVmr19QIAwFuyUEvNOGn 31vQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3aw1zzPFttGiixqkPWsGHzDixzvsuzy+40Vz1/ujh98=; b=dXYfNZ4IHnuG37GICGI/2jU4HIfWgH1hlN0a+tGTEd7G2jBqJf3Y4H+Ref4eotUo/e y9WR9mfewLjo7PP+m5Whaf3VK+E0zJbrfGUriBwMa6uM0B4g0buwLTSeUD27q7rczQPd qjByZQzc3SlNf1IFBsVqycD8rrlUd0pARJ2S3CWyxJg/KeDDBK+xAEpmX9JQcwHMsiAl LQiD4JBP4GoeAMjLg2A9Nx210fPrc35eWJ88QDXDEiAFWbqcsykdaYDeSIome9KsStcf wycc5XVAJnKw8txHJ1uot03UGdcU8oOgRy2drFwIdJi7LVm+769YqUxahjL61YXZRvmP PG1w==
X-Gm-Message-State: AOAM533v2/OfX4R0wwchW5NturwqzoQQ//WJGwNhrZNKLZ5CDf8OzP+R E/eW2k1IhN3SHUbJnky9wrnxeq15uzWt2+GGMybH7Q==
X-Google-Smtp-Source: ABdhPJySDbTGBPFrpiYYeA3nc6rACgX4Prg0QbIHKlYtmVfaoJPjVtK1rRj2rhEK/9ufQKkRkq/64H9MjG2Elia/4Wc=
X-Received: by 2002:a19:8252:: with SMTP id e79mr6847901lfd.103.1592221443397; Mon, 15 Jun 2020 04:44:03 -0700 (PDT)
MIME-Version: 1.0
References: <CALGR9obRjBSADN1KtKF6jvFVzNS1+JzaS0D0kCVKHKkd4sn+MQ@mail.gmail.com> <459C86F8-A989-4EF4-84DC-3568FF594F36@apple.com> <CANatvzwSpSHd7kZD-4tyMGkBJDdCBi6r_pLBvnaT8rrQy6SBHQ@mail.gmail.com> <CACMu3treK0m2mbpw9FebOjOcEed0bW-DbLbryHJH1DWAHoz+9g@mail.gmail.com> <CALGR9oZgE7ZfXdoYdUh9LUYC1fi8fMUyyTpvmV3GF7Z6Oxgg1g@mail.gmail.com> <20200609144428.GC22180@lubuntu> <CAJV+MGyuhxx=P6kZKktuREeq5pipZjxmwWP4jE_Sxhj_+krU2Q@mail.gmail.com> <CANatvzx_eg84V7UefOtSF+NHGHnTg7h-9n5bsRZRXxBqsaOkfQ@mail.gmail.com> <CACj=BEip6+7AunFsD=6qM5rsgrTfg6bRctOMu1gOe-KVjAW7Dw@mail.gmail.com> <CANatvzyv03VH9=+J=M2yY0EwCXp7HMWsXYaXOE=WYGDKBHdaVA@mail.gmail.com> <2C53D8AF-EFA8-42A3-9666-955A054468DB@greenbytes.de> <CACj=BEic2qzMXEfcsKS9CYnowChc-kMRjH66d3uKs+pqTz9Fug@mail.gmail.com> <4E0E8032-A903-46A2-A131-F1F4DE3CC037@greenbytes.de> <CACj=BEjOC-8S38U36Jw+Yb7yH_BZjxBkeLE6dLWH=8VMyBW80Q@mail.gmail.com> <ECF2C350-5D53-4E3B-9AAC-2F7E3FD4B528@greenbytes.de>
In-Reply-To: <ECF2C350-5D53-4E3B-9AAC-2F7E3FD4B528@greenbytes.de>
From: Yoav Weiss <yoav@yoav.ws>
Date: Mon, 15 Jun 2020 13:43:47 +0200
Message-ID: <CACj=BEh53ZWj1UV6tNDaWPiuHbmbVmkYimu_rdYcYm06dZJAAQ@mail.gmail.com>
To: Stefan Eissing <stefan.eissing@greenbytes.de>
Cc: Kazuho Oku <kazuhooku@gmail.com>, Patrick Meenan <patmeenan@gmail.com>, Lucas Pardue <lucaspardue.24.7@gmail.com>, HTTP Working Group <ietf-http-wg@w3.org>, =?UTF-8?Q?Bence_B=C3=A9ky?= <bnc@chromium.org>
Content-Type: multipart/alternative; boundary="00000000000089ab9905a81df18b"
Received-SPF: pass client-ip=2a00:1450:4864:20::12d; envelope-from=yoav@yoav.ws; helo=mail-lf1-x12d.google.com
X-W3C-Hub-Spam-Status: No, score=-8.9
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_DB=-1, W3C_IRA=-1, W3C_IRR=-3, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1jknXD-0000bG-9f e894ec82fb58afad27665fa15a8eeb5b
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Priority implementation complexity (was: Re: Extensible Priorities and Reprioritization)
Archived-At: <https://www.w3.org/mid/CACj=BEh53ZWj1UV6tNDaWPiuHbmbVmkYimu_rdYcYm06dZJAAQ@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/37765
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On Mon, Jun 15, 2020 at 1:18 PM Stefan Eissing <stefan.eissing@greenbytes.de>
wrote:

>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenweg 16
> 48155 Münster
> www.greenbytes.de
>
> > Am 15.06.2020 um 12:14 schrieb Yoav Weiss <yoav@yoav.ws>ws>:
> >
> >
> >
> > On Mon, Jun 15, 2020 at 11:03 AM Stefan Eissing <
> stefan.eissing@greenbytes.de> wrote:
> >
> > > Am 15.06.2020 um 10:28 schrieb Yoav Weiss <yoav@yoav.ws>ws>:
> > >
> > >
> > >
> > > On Mon, Jun 15, 2020 at 9:55 AM Stefan Eissing <
> stefan.eissing@greenbytes.de> wrote:
> > > > Am 11.06.2020 um 10:41 schrieb Kazuho Oku <kazuhooku@gmail.com>om>:
> > > >
> > > > That depends on how much clients would rely on reprioritization.
> Unlike H2 priorities, Extensible Priority does not have inter-stream
> dependencies. Therefore, losing *some* prioritization signals is less of an
> issue compared to H2 priorities.
> > > >
> > > > Assuming that reprioritization is used mostly for refining the
> initial priorities of a fraction of all the requests, I think there'd be
> benefit in defining reprioritization as an optional feature. Though I can
> see some might argue for not having reprioritization even as an optional
> feature unless there is proof that it would be useful.
> > >
> > >
> > > > We should decide if reprioritization is good or bad, based on as
> much data as we can pull, and make sure it's implemented only if we see
> benefits for it in some cases, and then make sure it's only used in those
> cases.
> > >
> > > When thinking about priority implementations, I recommend thinking
> about a H3 reverse proxy in front of a legacy H1 server. Assume limited
> memory, disk space and backend connections.
> > >
> > > (Re-)prioritization in H2 works well for flow control, among the
> streams that have response data to send. Priorities can play a part in
> server scheduling, but
> > > it's more tricky. By "scheduling" I mean that the server has to pick
> one among the opened streams for which it wants to compute a response for.
> This is often impossible to re-prioritize afterwards (e.g. suicidal for a
> server implementation).
> > >
> > > Can you expand on why it is "suicidal"?
> >
> > It is tricky to obey re-prioritizations to the letter, managing
> memory+backend connections and protecting the infrastructure against DoS
> attacks. The reality is that there are limited resources and a server is
> expected to protect those. It's a (pun intended) top priority.
> >
> > Another priority topping the streams is the concept of fairness between
> connections. In Apache httpd, the resources to process h2 streams are
> foremost shared evenly between connections.
> >
> > That makes sense. Would re-prioritization of specific streams somehow
> require to change that?
> >
> > The share a connection gets is then allocated to streams based on
> current h2 priority settings. Any change after that will "only" affect the
> downstream DATA allocation.
> >
> > I *think* this makes sense as well, assuming that by "downstream" you
> mean "future". Is that what you meant? Or am I missing something?
> >
> > Also, the number of "active" streams on a connection is dynamic. It will
> start relatively small and grow if the connection is well behaving, shrink
> if it is not. That one of the reasons that Apache was only partially
> vulnerable to a single issue on the Netflix h2 cve list last year (the
> other being nghttp2).
> >
> > tl;dr
> >
> > By "suicidal" I mean a server failing the task of process thousands of
> connections in a consistent and fair manner.
> >
> > Apologies if I'm being daft, but I still don't understand how (internal
> to a connection) stream reprioritization impacts cross-connection fairness.
>
> *fails to imagine Yoav as being daft*
>
:)

Thanks for outlining the server-side processing!


> A server with active connections and workers. For simplicity, assume that
> each ongoing request allocates a worker.
> - all workers are busy
> - re-prio arrives and makes a stream A, being processed, depend on a
> stream B which has not been assigned a worker yet.
>

OK, I now understand that this can be concerning.
IIUC, this part is solved by with Extensible Priorities (because there's no
dependency tree).

Lucas, Kazuho - can you confirm?


> - ideally, the server would freeze the processing of A and assign the
> resources to B.
> - however re-allocating the resources is often not possible  (Imagine a
> CGI process running or a backend HTTP/1.1 or uWSGI connection.)
> - the server can only suspend the worker or continue processing, ignoring
> the dependency.
> - a suspended worker is very undesirable and a possible victim of a
> slow-loris attack
> - To make this suspending less sever, the server would need to make
> processing of stream B very important. To unblock it quickly again. This is
> then where unfairness comes in.
>
> The safe option therefore is to continue processing stream A and ignore
> the dependency on B. Thus, priorities are only relevant:
> 1. when the next stream to process on a connection is selected
> 2. when size/number of DATA frames to send is allocated on a connection
> between all streams that want to send
>
> (Reality is often not quite as bad as I described: when static file/cache
> resources are served for example, a worker often just does the lookup,
> producing a file handle very quickly. A connection easily juggles a number
> of file handles to stream out according to priorities and stalling one file
> on another comes at basically no risk and cost.)
>
> Now, this is for H2 priorities. I don't know enough about QUIC priorities
> to have an opinion on the proposals. Just wanted to point out that servers
> see the world a little different than clients. ;)
>

I checked and it seems like Chromium does indeed change the parent
dependency as part of reprioritization. If the scenario you outlined is a
problem in practice, we should discuss ways to avoid doing that with H2
priorities.


>
> Cheers, Stefan
>
>
> > >
> > >
> > > If we would do H2 a second time, my idea would be to signal priorities
> in the HTTP request in a connection header and use this in the H2 frame
> layer to allocate DATA space on the downlink. Leave out changing priorities
> on a request already started. Let the client use its window sizes if it
> feels the need.
> > >
> > > Cheers, Stefan (lurking)
> >
>
>