Re: Structured request headers deployment issues

Yoav Weiss <> Fri, 19 June 2020 08:41 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 317973A0833 for <>; Fri, 19 Jun 2020 01:41:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.648
X-Spam-Status: No, score=-2.648 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id YWHPFbJHlvsN for <>; Fri, 19 Jun 2020 01:41:53 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 8B5893A082F for <>; Fri, 19 Jun 2020 01:41:52 -0700 (PDT)
Received: from lists by with local (Exim 4.92) (envelope-from <>) id 1jmCXr-0001z6-KX for; Fri, 19 Jun 2020 08:38:43 +0000
Resent-Date: Fri, 19 Jun 2020 08:38:43 +0000
Resent-Message-Id: <>
Received: from ([]) by with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <>) id 1jmCXo-0001yL-MO for; Fri, 19 Jun 2020 08:38:40 +0000
Received: from ([2a00:1450:4864:20::22b]) by with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from <>) id 1jmCXl-0003eC-Hb for; Fri, 19 Jun 2020 08:38:40 +0000
Received: by with SMTP id x18so10539593lji.1 for <>; Fri, 19 Jun 2020 01:38:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=364QeniKCc1eTHFnIf74dCn1PhKPzUowUtNV3rR8sX0=; b=bmzjwnePYS4V/r5EGOTVPYfMuRP7usSuPeP5HIZ6NldgjMLIx3KMfD1Mj4ROiwhpEa kdePhuINpJ0K32exwjshYyi2wiImOfEFmCTf8bB561m982WkQQTlan2c5stYyR1l9kiz GvFVO1bAzDbC/9lYTf15osRVCQXjSYuh92smBKcNMo8hDb24X6xFGDTOIybfy2rEsAre di04zQPSvhHlg8Olyn3DBj4UcGPHpbLLT3Hq164e2kCZYnbJUnIDcMpc0BT8lcQq21Ln 5xJJbJUboIs8d2n3ZLHOlwvMM7cMh8CD5+GwNdJGnbcUhM01PhxV0hv+vbVNpgM1UKPz b/aQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=364QeniKCc1eTHFnIf74dCn1PhKPzUowUtNV3rR8sX0=; b=a9rwRKkg9UspgXHaEHtefHhyzbbU2owbLoVOjzCfhE6xGSpxEzopz14Y377bTfDiil gj5tkwUQ9zZEBjYLctuyw4/x2BSx4u1dn8RnGnKpKIbmCP26kuB75VpMP4E7Q9pPrz6T ItIwxCrIaLg/KAM9rB2HO39nJ4I3RdqsFhbYm11Rv4LbqynIO5hd+qi0LkX1vslfX0JH BDxynHBdfS95qlzneihyS2hBT+IUjmV/jld7hCU0joqzIXQINWwOHB5K8jEJcQqjyJEI 4ruk8KzrPstO29ZiVK6NvzYSOS+lbrt4ae/ppis+AYAkLBH7pL46RWSYCh2SSWD8P6uB yIrg==
X-Gm-Message-State: AOAM530/uatu5xlEMjRXKCEDVstCPRZ+3058b/BMGauQiZO1TATs48Dy 9WHYfT/gqwc0LxkmpgqkefxHWZxpVqdvR4nXgb3xvQ==
X-Google-Smtp-Source: ABdhPJwlTBpwNBgjwh1XO33BaIxgZlBvqHsoUHMhQwpwlIYTyi49UPXMHJKwkWb4h638ARZ2grVVtanJMrOEyaTdFo0=
X-Received: by 2002:a2e:b0f9:: with SMTP id h25mr1306077ljl.18.1592555905385; Fri, 19 Jun 2020 01:38:25 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <>
In-Reply-To: <>
From: Yoav Weiss <>
Date: Fri, 19 Jun 2020 10:38:09 +0200
Message-ID: <>
To: Mike West <>
Cc: Mark Nottingham <>, " Group" <>, Tommy Pauly <>, Ilya Grigorik <>
Content-Type: multipart/alternative; boundary="000000000000069a1c05a86bd199"
Received-SPF: pass client-ip=2a00:1450:4864:20::22b;;
X-W3C-Hub-Spam-Status: No, score=-8.9
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_DB=-1, W3C_IRA=-1, W3C_IRR=-3, W3C_WL=-1
X-W3C-Scan-Sig: 1jmCXl-0003eC-Hb 905857156329dc397cb0b1daafcdea6a
Subject: Re: Structured request headers deployment issues
Archived-At: <>
X-Mailing-List: <> archive/latest/37797
Precedence: list
List-Id: <>
List-Help: <>
List-Post: <>
List-Unsubscribe: <>

On Fri, Jun 19, 2020 at 9:03 AM Mike West <> wrote:

> On Fri, Jun 19, 2020 at 7:13 AM Mark Nottingham <> wrote:
>> > On 16 Jun 2020, at 8:15 am, Yoav Weiss <> wrote:
>> >
>> > Hey all,
>> >
>> > Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent
>> Client Hints enabled by default, which is using Structured Headers.
>> >
>> > As a result of that, we found multiple sites which seem to have a
>> somewhat allergic reaction to the presence of certain characters (that are
>> part of the SH format) in request values.
>> > While each site in question is different (in what appears to be coming
>> from different stacks), we've seen sites that reject requests with quotes,
>> question marks or equals signs in them.
>> > It's still early, so it's hard to know how widespread the issue is, but
>> we seem to be adding sites to the list at a faster pace than the pace of
>> removing fixed ones from it.
> Yoav, have we experimented at all with alternate spellings? Do
> single-quotes have the same impact as double-quotes, for example? Are other
> (uglier) non-alphanumeric delimiters less likely to cause trouble? There's
> a limited appetite for breakage, and it might be worth exploring alternate
> aesthetics rather than forcing through the particular spelling we like the
> most.

We have not. But given the fact that we've seen issues with the use of `"`,
`?` and `=` separately, I'm somewhat assuming that those separate
implementations are using an allow-list approach.
With that said, I can try to throw various values at those known-to-break
servers and see if I can come up with an alternative spelling that would
work with them. It won't give us any guarantees that these spellings would
work at scale, but it might be a good starting point...

> > So, I wanted to give this group a heads-up on that front, and maybe get
>> folks' opinions regarding possible things we could do on that front, other
>> than outreach and waiting for said sites to fix themselves.
>> AIUI these aren't new; e.g., IIRC quite a few months ago Chrome
>> encountered several Austrian sites that had this problem, traced back to a
>> local(?) WAF vendor there. I believe that's been corrected since, after
>> reaching out to them.
> I think you might be referring to some conversations around the
> (since-renamed) `Sec-Metadata` header in 2018, like those captured in
> Personally, I think that outreach and waiting is the right approach; if
>> browsers consistently send these headers, they'll adapt, and the numbers
>> are still relatively small -- or at least small enough that it's not likely
>> the numbers will be reduced if the syntax is changed (due to _other_ WAFs'
>> opinions about what a "good" request is).
> The flip-side of this is that we break sites users rely upon. That's tough
> to do at scale. In this case in particular, I think I agree with the
> underlying assertion that we'd like this syntax to work, and it seems
> reasonable to me to roll it out to a small percentage of stable users. That
> said, if we end up breaking the internet for even a small percentage of
> users, it doesn't seem like a good idea to hold them hostage in the hopes
> of increasing pressure on WAF vendors.
> My expectation is that we'd roll this out to a small percentage of stable,
> see a spike in bug reports (especially from enterprise folks who are likely
> to have WAF deployments, unlikely to broadly deploy beta channels, and
> unlikely to choose to enable usage statistics or histograms), see a
> corresponding spike in error codes for top-level navigations, and roll it
> back.
> I think it might be reasonable to explore alternate spellings at the same
> time, perhaps with some A/B testing to evaluate how well each weaves its
> way through middleware.
> Related, we're also seeing more examples WAFs limiting how we can evolve
>> the protocol (e.g., <>).
>> There's been a bit of background chatter about writing something about this
>> and creating better communication with that community; I'm not sure what
>> that will look like yet, but if anyone has ideas or is interested, please
>> say so.
> I am interested!
> -mike