Re: Structured request headers deployment issues

Mark Nottingham <mnot@mnot.net> Fri, 19 June 2020 05:16 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 86D803A0E3F for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Thu, 18 Jun 2020 22:16:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.749
X-Spam-Level:
X-Spam-Status: No, score=-2.749 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=mnot.net header.b=dx9AhABJ; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=AP97+YYQ
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rStPfdcwKMRQ for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Thu, 18 Jun 2020 22:16:26 -0700 (PDT)
Received: from lyra.w3.org (lyra.w3.org [128.30.52.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 923CF3A0E33 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Thu, 18 Jun 2020 22:16:26 -0700 (PDT)
Received: from lists by lyra.w3.org with local (Exim 4.92) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1jm9LQ-00063Z-8Q for ietf-http-wg-dist@listhub.w3.org; Fri, 19 Jun 2020 05:13:40 +0000
Resent-Date: Fri, 19 Jun 2020 05:13:40 +0000
Resent-Message-Id: <E1jm9LQ-00063Z-8Q@lyra.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by lyra.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <mnot@mnot.net>) id 1jm9LM-00062h-Nx for ietf-http-wg@listhub.w3.org; Fri, 19 Jun 2020 05:13:36 +0000
Received: from out2-smtp.messagingengine.com ([66.111.4.26]) by mimas.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <mnot@mnot.net>) id 1jm9LK-0007fs-OU for ietf-http-wg@w3.org; Fri, 19 Jun 2020 05:13:36 +0000
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id BE4895C014F; Fri, 19 Jun 2020 01:13:19 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 19 Jun 2020 01:13:19 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mnot.net; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm3; bh=j qvlvCO9gmVGQHhA+QTH7HxdQlPuR4suODLcXmX9CD0=; b=dx9AhABJDePMKz+// r4s7SuMbOByUXscPRnF15TO2tABhfQa2tSeNiaRNnBJNRJr4o0t5IGJP8KUeJXM+ NF+e2eb4bHPzgW244/cj/a5xarJLIUgibVqM24zMg0fCNPsf7ehxirG7LhThxVC7 7Fj2ALFrBnGBOcwjUIPWDH+lZPuKf2JPbPBlUU70skVzb2iGmc93BwMgTQF+d7Im xFbIiLI1UeCFUOwW7Crk53VtJvJI1KwjppYcAjlmuR4zO6S1kZfWRKg4neghEP6o Cl5hrpq8vSovfAD6Jv9TKeXLkTZ7YJvA9HJnPAAgTOWnIFDlteBJZth32LJNlFbF VD4cg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=jqvlvCO9gmVGQHhA+QTH7HxdQlPuR4suODLcXmX9C D0=; b=AP97+YYQZ1Qi7TmzW9vAmKXW0tGCX/CD+Mo9+5DCkFHcl6rnuA7fI3jw/ c43GHmi5QA5tDs96Ksnj7PwtWT0y8o4hwOfVMJdQdJQqGtrTukrHU/FGOI1IKysQ hbUBTCFilwAoQA9pYJyb6wFWch5qab/3qW3jtMgieKH918vKpCupA1o+WAsLoq3I biGT7kIMgeFBhOLbwgary75zkpximsFZDaKP4v3V50NQEYd0MGm8AjCydztCyeGQ oXa2cswLwziVWrtPGSCyLUblHfeK8ePYIIRr+3JS1hkpUDZNqj6xK9vht1hEM743 NEs7l5u0MOTJCYw66GUzIFEbvQ1EA==
X-ME-Sender: <xms:bknsXqRiv3CxMAdcdU4PHKJmt_HvTrOu59jNdOLKJQjqtNp19yXvnw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrudejhedgleefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdludefmdenucfjughrpegtggfuhfgjfffgkfhfvffosehtqhhmtdhh tddvnecuhfhrohhmpeforghrkhcupfhothhtihhnghhhrghmuceomhhnohhtsehmnhhoth drnhgvtheqnecuggftrfgrthhtvghrnhepleffvdeuveffffekgefgffeugeehleekkeet jeelhfelkeevkeduieeivedvtefgnecuffhomhgrihhnpehgihhthhhusgdrtghomhdpmh hnohhtrdhnvghtnecukfhppeduudelrddujedrudehkedrvdehudenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmnhhothesmhhnohhtrdhnvg ht
X-ME-Proxy: <xmx:bknsXvxzQmNyLZHES0JdSxCNr9javlqQ3rwY1Jz4TYBS5NF2Hw2xBg> <xmx:bknsXn0iMGPrXnGQsIyGpDPKFIpnNDfqlB15CyIuut-o-kJg8kPa-g> <xmx:bknsXmBpmvDJVyo0_ZlrQHvzezjDXQkshkKEdEUNNVtegOqgZM5YPQ> <xmx:b0nsXjeZltNRzs7AXdvfn0VJ1QYxfN7a3XXSd5UF4XRmhK4LmaQyTg>
Received: from macbook-air.mnot.net (119-17-158-251.77119e.mel.static.aussiebb.net [119.17.158.251]) by mail.messagingengine.com (Postfix) with ESMTPA id E48F4328005D; Fri, 19 Jun 2020 01:13:16 -0400 (EDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
From: Mark Nottingham <mnot@mnot.net>
In-Reply-To: <CACj=BEiT7GnKeS_2wFK8jL0jUFtFYoX-wvXnSsPO4nYJ5P=2bQ@mail.gmail.com>
Date: Fri, 19 Jun 2020 15:13:13 +1000
Cc: "ietf-http-wg@w3.org Group" <ietf-http-wg@w3.org>, Tommy Pauly <tpauly@apple.com>, Ilya Grigorik <igrigorik@gmail.com>, Mike West <mkwst@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <36626BBC-F97C-4A7A-8F1F-E3E9FBA920EA@mnot.net>
References: <CACj=BEiT7GnKeS_2wFK8jL0jUFtFYoX-wvXnSsPO4nYJ5P=2bQ@mail.gmail.com>
To: Yoav Weiss <yoav@yoav.ws>
X-Mailer: Apple Mail (2.3608.80.23.2.2)
Received-SPF: pass client-ip=66.111.4.26; envelope-from=mnot@mnot.net; helo=out2-smtp.messagingengine.com
X-W3C-Hub-Spam-Status: No, score=-9.8
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_DB=-1, W3C_IRA=-1, W3C_IRR=-3, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1jm9LK-0007fs-OU 7618bfb1a58457bd57f0320ca1fc1d8a
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Structured request headers deployment issues
Archived-At: <https://www.w3.org/mid/36626BBC-F97C-4A7A-8F1F-E3E9FBA920EA@mnot.net>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/37795
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

Hey Yoav,

> On 16 Jun 2020, at 8:15 am, Yoav Weiss <yoav@yoav.ws> wrote:
> 
> Hey all,
> 
> Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent Client Hints enabled by default, which is using Structured Headers.
> 
> As a result of that, we found multiple sites which seem to have a somewhat allergic reaction to the presence of certain characters (that are part of the SH format) in request values. 
> While each site in question is different (in what appears to be coming from different stacks), we've seen sites that reject requests with quotes, question marks or equals signs in them.
> It's still early, so it's hard to know how widespread the issue is, but we seem to be adding sites to the list at a faster pace than the pace of removing fixed ones from it.
> 
> So, I wanted to give this group a heads-up on that front, and maybe get folks' opinions regarding possible things we could do on that front, other than outreach and waiting for said sites to fix themselves.

AIUI these aren't new; e.g., IIRC quite a few months ago Chrome encountered several Austrian sites that had this problem, traced back to a local(?) WAF vendor there. I believe that's been corrected since, after reaching out to them.

Personally, I think that outreach and waiting is the right approach; if browsers consistently send these headers, they'll adapt, and the numbers are still relatively small -- or at least small enough that it's not likely the numbers will be reduced if the syntax is changed (due to _other_ WAFs' opinions about what a "good" request is).

Also, if we get these headers through, it seems like it would give us good protection (of a sort) for future Structured request headers.

Related, we're also seeing more examples WAFs limiting how we can evolve the protocol (e.g., <https://github.com/coreruleset/coreruleset/pull/1777>). There's been a bit of background chatter about writing something about this and creating better communication with that community; I'm not sure what that will look like yet, but if anyone has ideas or is interested, please say so.

Cheers,

--
Mark Nottingham   https://www.mnot.net/