Re: Structured request headers deployment issues

Yoav Weiss <yoav@yoav.ws> Fri, 19 June 2020 10:18 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 436F33A094E for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Fri, 19 Jun 2020 03:18:38 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.648
X-Spam-Level:
X-Spam-Status: No, score=-2.648 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=yoav-ws.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BJRZb2GSh9GX for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Fri, 19 Jun 2020 03:18:35 -0700 (PDT)
Received: from lyra.w3.org (lyra.w3.org [128.30.52.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B2CAB3A094A for <httpbisa-archive-bis2Juki@lists.ietf.org>; Fri, 19 Jun 2020 03:18:35 -0700 (PDT)
Received: from lists by lyra.w3.org with local (Exim 4.92) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1jmE3d-00040i-F1 for ietf-http-wg-dist@listhub.w3.org; Fri, 19 Jun 2020 10:15:37 +0000
Resent-Date: Fri, 19 Jun 2020 10:15:37 +0000
Resent-Message-Id: <E1jmE3d-00040i-F1@lyra.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by lyra.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <yoav@yoav.ws>) id 1jmE3b-0003zs-0S for ietf-http-wg@listhub.w3.org; Fri, 19 Jun 2020 10:15:35 +0000
Received: from mail-lj1-x234.google.com ([2a00:1450:4864:20::234]) by mimas.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from <yoav@yoav.ws>) id 1jmE3U-0007cn-J6 for ietf-http-wg@w3.org; Fri, 19 Jun 2020 10:15:34 +0000
Received: by mail-lj1-x234.google.com with SMTP id e4so10851376ljn.4 for <ietf-http-wg@w3.org>; Fri, 19 Jun 2020 03:15:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yoav-ws.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=pmOozD69Vy3qPCCRLo8qUOI5+qdufSisKJ4U1Y4Z47A=; b=OvmHDHo99c05nsAPqxKOCn0y/Au5cefz4oP+hEo1MMU1ucaWbK4fPsq7Y5A1jyF8l4 UkTCXmLl1TWZzPQz8JF53fylsd+22bK9qoVTMWv3Fd+9/xrViXx9wUEXbbg02KBQad1D Jmg4Ny1D0X0dvmi6eGqSno0YEMvzUZT8kl5f88CJ5iCniTK6wP1MjLfdV/oRPLVigWyd 1zVvDlQQWWMNqe9yFRIdQA0VQvZ+zgUgljfhQkfAYVm5x+YKmHynMhSY2yIfrkufF60I D3e/KRi+M3zGhpJvZ2OVpNuTaqYeEfe9bfkQuWh8zYKW1fhQChFh5IZn0roPIWZGqmtJ uggQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pmOozD69Vy3qPCCRLo8qUOI5+qdufSisKJ4U1Y4Z47A=; b=oyqxjHCxO+O6YPKsMJROCBfXOUIrKGSBBKmJpXOguvXjPRxQgAIRmRVfdq0bjMOtkG 0c0FtaJ0vuTOLe/NvuG6qYoenbGXTW9HQhnhuSq2XHU1DbicZijCvaLVq0DqTUa96dPI +hGqAHg41hDPiC9graCqjtZZQXDWCs7cloEKUO7E49ECAdihDsCE1plLwvqmrpgwJVNV sCo2U081BsL7a0XKUdL/4PjIDHDMteh0bzoJdf8d7oa5MXWcD64wihdPonjLPsooapkr pvF65bJtoNaEHN3tzYzF5/DfeZdgVmkiXKsyShVsMakQ3oNk2NBE7J1eI8bbt0Jry9wy wPsQ==
X-Gm-Message-State: AOAM530JX7WrWRpiuYxifYAcsiADLoEUee6bAiYt4D3SFHCrt+jCiODb cNdJXZMcCdw1cvOPr5lHScucfGCoD9GnfeBuWgVQ6Q==
X-Google-Smtp-Source: ABdhPJz7F08EzQOfmqLaYUqFu6v2JSNJeZfSNyQ/eyquXhqh4Tj/OAS/zXKK3d57UNCfrRCdqiJveXG79dsuvDLU5nk=
X-Received: by 2002:a2e:7105:: with SMTP id m5mr1375685ljc.79.1592561716796; Fri, 19 Jun 2020 03:15:16 -0700 (PDT)
MIME-Version: 1.0
References: <CACj=BEiT7GnKeS_2wFK8jL0jUFtFYoX-wvXnSsPO4nYJ5P=2bQ@mail.gmail.com> <36626BBC-F97C-4A7A-8F1F-E3E9FBA920EA@mnot.net> <CAKXHy=fbcL4XEf8NeQ8kW-+d4c8Q6EwO3U0Ara626FX7hmY07A@mail.gmail.com> <CACj=BEgSJEZ5dZjh7F98zKcT8a1nfko8i03_e=+MntWCRKggUw@mail.gmail.com>
In-Reply-To: <CACj=BEgSJEZ5dZjh7F98zKcT8a1nfko8i03_e=+MntWCRKggUw@mail.gmail.com>
From: Yoav Weiss <yoav@yoav.ws>
Date: Fri, 19 Jun 2020 12:15:00 +0200
Message-ID: <CACj=BEht-M7cmsiXYwb_A-o_3pHF3x9O1CUj2ffxeERfz_690w@mail.gmail.com>
To: Mike West <mkwst@google.com>
Cc: Mark Nottingham <mnot@mnot.net>, "ietf-http-wg@w3.org Group" <ietf-http-wg@w3.org>, Tommy Pauly <tpauly@apple.com>, Ilya Grigorik <igrigorik@gmail.com>
Content-Type: multipart/alternative; boundary="00000000000069bba805a86d2b69"
Received-SPF: pass client-ip=2a00:1450:4864:20::234; envelope-from=yoav@yoav.ws; helo=mail-lj1-x234.google.com
X-W3C-Hub-Spam-Status: No, score=-8.9
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_DB=-1, W3C_IRA=-1, W3C_IRR=-3, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1jmE3U-0007cn-J6 73bfe595f44ead37dcae566cbd181c50
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Structured request headers deployment issues
Archived-At: <https://www.w3.org/mid/CACj=BEht-M7cmsiXYwb_A-o_3pHF3x9O1CUj2ffxeERfz_690w@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/37798
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On Fri, Jun 19, 2020 at 10:38 AM Yoav Weiss <yoav@yoav.ws> wrote:

>
>
> On Fri, Jun 19, 2020 at 9:03 AM Mike West <mkwst@google.com> wrote:
>
>> On Fri, Jun 19, 2020 at 7:13 AM Mark Nottingham <mnot@mnot.net> wrote:
>>
>>> > On 16 Jun 2020, at 8:15 am, Yoav Weiss <yoav@yoav.ws> wrote:
>>> >
>>> > Hey all,
>>> >
>>> > Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent
>>> Client Hints enabled by default, which is using Structured Headers.
>>> >
>>> > As a result of that, we found multiple sites which seem to have a
>>> somewhat allergic reaction to the presence of certain characters (that are
>>> part of the SH format) in request values.
>>> > While each site in question is different (in what appears to be coming
>>> from different stacks), we've seen sites that reject requests with quotes,
>>> question marks or equals signs in them.
>>> > It's still early, so it's hard to know how widespread the issue is,
>>> but we seem to be adding sites to the list at a faster pace than the pace
>>> of removing fixed ones from it.
>>>
>>
>> Yoav, have we experimented at all with alternate spellings? Do
>> single-quotes have the same impact as double-quotes, for example? Are other
>> (uglier) non-alphanumeric delimiters less likely to cause trouble? There's
>> a limited appetite for breakage, and it might be worth exploring alternate
>> aesthetics rather than forcing through the particular spelling we like the
>> most.
>>
>
> We have not. But given the fact that we've seen issues with the use of
> `"`, `?` and `=` separately, I'm somewhat assuming that those separate
> implementations are using an allow-list approach.
> With that said, I can try to throw various values at those known-to-break
> servers and see if I can come up with an alternative spelling that would
> work with them. It won't give us any guarantees that these spellings would
> work at scale, but it might be a good starting point...
>

I played around with trying to find a safe sete of chars that would work as
delimiters on the set of sites we know are broken. I have failed
<https://bugs.chromium.org/p/chromium/issues/detail?id=1091285#c10> to
find such a subset :/


>
>>
>> > So, I wanted to give this group a heads-up on that front, and maybe get
>>> folks' opinions regarding possible things we could do on that front, other
>>> than outreach and waiting for said sites to fix themselves.
>>>
>>> AIUI these aren't new; e.g., IIRC quite a few months ago Chrome
>>> encountered several Austrian sites that had this problem, traced back to a
>>> local(?) WAF vendor there. I believe that's been corrected since, after
>>> reaching out to them.
>>>
>>
>> I think you might be referring to some conversations around the
>> (since-renamed) `Sec-Metadata` header in 2018, like those captured in
>> https://bugs.chromium.org/p/chromium/issues/detail?id=861678#c11.
>>
>> Personally, I think that outreach and waiting is the right approach; if
>>> browsers consistently send these headers, they'll adapt, and the numbers
>>> are still relatively small -- or at least small enough that it's not likely
>>> the numbers will be reduced if the syntax is changed (due to _other_ WAFs'
>>> opinions about what a "good" request is).
>>>
>>
>> The flip-side of this is that we break sites users rely upon. That's
>> tough to do at scale. In this case in particular, I think I agree with the
>> underlying assertion that we'd like this syntax to work, and it seems
>> reasonable to me to roll it out to a small percentage of stable users. That
>> said, if we end up breaking the internet for even a small percentage of
>> users, it doesn't seem like a good idea to hold them hostage in the hopes
>> of increasing pressure on WAF vendors.
>>
>> My expectation is that we'd roll this out to a small percentage of
>> stable, see a spike in bug reports (especially from enterprise folks who
>> are likely to have WAF deployments, unlikely to broadly deploy beta
>> channels, and unlikely to choose to enable usage statistics or histograms),
>> see a corresponding spike in error codes for top-level navigations, and
>> roll it back.
>>
>> I think it might be reasonable to explore alternate spellings at the same
>> time, perhaps with some A/B testing to evaluate how well each weaves its
>> way through middleware.
>>
>> Related, we're also seeing more examples WAFs limiting how we can evolve
>>> the protocol (e.g., <
>>> https://github.com/coreruleset/coreruleset/pull/1777>)t;). There's been a
>>> bit of background chatter about writing something about this and creating
>>> better communication with that community; I'm not sure what that will look
>>> like yet, but if anyone has ideas or is interested, please say so.
>>>
>>
>> I am interested!
>>
>> -mike
>>
>