Re: Structured request headers deployment issues

[Mark Nottingham <[e-mail]>]

Hey Yoav,

> On 16 Jun 2020, at 8:15 am, Yoav Weiss <[e-mail]> wrote:
> 
> Hey all,
> 
> Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent Client Hints enabled by default, which is using Structured Headers.
> 
> As a result of that, we found multiple sites which seem to have a somewhat allergic reaction to the presence of certain characters (that are part of the SH format) in request values. 
> While each site in question is different (in what appears to be coming from different stacks), we've seen sites that reject requests with quotes, question marks or equals signs in them.
> It's still early, so it's hard to know how widespread the issue is, but we seem to be adding sites to the list at a faster pace than the pace of removing fixed ones from it.
> 
> So, I wanted to give this group a heads-up on that front, and maybe get folks' opinions regarding possible things we could do on that front, other than outreach and waiting for said sites to fix themselves.

AIUI these aren't new; e.g., IIRC quite a few months ago Chrome encountered several Austrian sites that had this problem, traced back to a local(?) WAF vendor there. I believe that's been corrected since, after reaching out to them.

Personally, I think that outreach and waiting is the right approach; if browsers consistently send these headers, they'll adapt, and the numbers are still relatively small -- or at least small enough that it's not likely the numbers will be reduced if the syntax is changed (due to _other_ WAFs' opinions about what a "good" request is).

Also, if we get these headers through, it seems like it would give us good protection (of a sort) for future Structured request headers.

Related, we're also seeing more examples WAFs limiting how we can evolve the protocol (e.g., <https://github.com/coreruleset/coreruleset/pull/1777>). There's been a bit of background chatter about writing something about this and creating better communication with that community; I'm not sure what that will look like yet, but if anyone has ideas or is interested, please say so.

Cheers,

--
Mark Nottingham   https://www.mnot.net/