Re: [Http-grease] Brief review of Greasing HTTP (draft-nottingham-http-grease-00)

[Mark Nottingham <[e-mail]>]

Thanks for the feedback, Christian!

> On 8 Oct 2020, at 2:27 am, Christian Folini <[e-mail]> wrote:
> 
> Thank you Mark!
> 
> I meant to respond to your September message earlier - but then I did not.
> 
> Meanwhile, I have read through your updated draft of the "Greasing HTTP"
> document. It's stripped down now and I see it as lighter and probably more
> adequate until we have earned some experience (without laying down the
> process in detail).
> 
> I'm not sure about the following point:
> 
> "For example, it is increasingly common for Web Application Firewalls (WAFs),
> bot detection services and similar components to reject HTTP requests that
> contain an unrecognised HTTP header field;"
> 
> Is there evidence for that? I may be biased because the OWASP CRS WAF project
> that I co-run does not do this. But I also have not heard of any other WAF or
> CDN implementing such a filter by default.

I think it's been encountered, but rarely. I'm happy to refocus this on unknown headers whose value contain certain characters or strings, which we see a lot more.


> Defining the format of known header fields is a different matter though. There
> are definitely filters that define how the WAFs think that - e.g. - a
> Content-Type header is supposed to look like.

Yes - for example, if certain headers can contain parameters, we should probably grease that.


> I like the day how the http-grease mailinglist is meant to become the go-to
> place for greasing announcements. That makes a lot of sense from my point of
> view.

Great.

> One way where WAFs could make it easier is providing demo sites, where people
> (-> greasers!) can try out requests. This has been on the to-do list for our
> CRS project quite some time.

That would be helpful!

I'm going to make some edits based upon the above and submit a -00 so the WG can begin discussing it.

Thanks again,




> 
> Best regards,
> 
> Christian
> 
> 
> 
> On Tue, Sep 08, 2020 at 04:01:31PM +1000, Mark Nottingham wrote:
>> Hey Christian (et al),
>> 
>> Thanks for the review; I want to incorporate your (excellent) feedback, but before that, I think the draft probably needs a bigger rewrite, because I've been thinking about it and talking to people about it in the background.
>> 
>> One of the big takeaways from those discussions was that it's unlikely implementations will be able to coordinate greasing in a meaningful way, because of how different their shipping schedules and distribution models can be. Having said that, most of the folks I talked to seemed to recognise the necessity of doing something here, and were willing (even eager) to help out. Because of various constraints, however, it's likely that any greasing we'll see will be mostly in nightly and beta releases for the immediate future.
>> 
>> So, I suspect the draft currently over-states the value of coordinated action between the clients; while it's valuable for a client to not be the only one 'sticking their neck out', there doesn't need to be such a high degree of choreography between them (so we can probably drop the process bits).
>> 
>> What I think we need is:
>> 
>> 1. Data
>> 
>> More understanding of what and how extension points are starting to 'rust' -- e.g., what characters or strings cause requests to be rejected?
>> 
>> The obvious way to do this is spidering live sites, but that's a pretty involved project, with less-than-certain results (especially if WAFs etc. are selective about the request URLs they apply filters to).
>> 
>> Another way to be more effective with the resources we have would be to add some rigour to how implementations collect bug reports. E.g., if browsers had a tag / label in their bug queues for ossification, it should be simple to combine them into a single data feed.
>> 
>> 2. Air cover for greasing
>> 
>> Implementations that start greasing need something that they can point to to explain what they're doing, why they're doing it, and to help give them some moral authority (ideally). A HTTP WG document is the logical thing here, I think.
>> 
>> 3. Lightweight coordination
>> 
>> So that implementations can tell folks what they're greasing, both so that greasers can align efforts where it makes sense, and greasees (?) get a heads-up.
>> 
>> This mailing list should be able to serve that function.
>> 
>> 
>> Does that make sense to everyone, and is anything missing? I'm willing to rewrite the draft along the 'air cover' theme, and if implementations tag their ossification bugs, I'd be willing to try to write tools to take advantage of that.
>> 
>> Cheers,
>> 
>> 
>> 
>>> On 12 Aug 2020, at 12:13 am, Christian Folini <[e-mail]> wrote:
>>> 
>>> Hi there,
>>> 
>>> The Summer holidays kept me from reviewing this draft document earlier, but
>>> here we go.
>>> 
>>> Link and version: https://mnot.github.io/I-D/http-grease/ v00, July 7, 2020.
>>> 
>>> 
>>> I'm taking the perspective of a co-lead of the OWASP ModSecurity Core Rule Set
>>> project (CRS), a popular WAF rule set used by quite a few vendors for their
>>> offerings.
>>> 
>>> Intro:
>>> 
>>> The high-flying explanation of what "greasing" actually means, could be a bit
>>> more explicit. Maybe even a section "What is greasing?"  It would help with
>>> the readability (-> and consequently adoption) of the document.
>>> 
>>> I read the doc several times and I got it the 2nd time I read it. But it was
>>> all a bit puzzling at first sight.
>>> 
>>> 
>>> What to Grease?
>>> 
>>> The doc explains how HTTP response headers are greased daily by the various
>>> server implementations themselves.
>>> 
>>> The argumentation is a bit thin in this regard, since there is an equally
>>> large number of custom HTTP request headers being sent across the wire. One
>>> would need to explain why the situation with these request headers is
>>> different from the situation with the response headers.
>>> 
>>> 
>>> 
>>> How to Grease?
>>> 
>>> There is this passage "an implementation determined to control input values
>>> can anticipate grease values and allow them, ...". I think it is very
>>> important to see and anticipate this.
>>> 
>>> This is especially true when you read it together with the non-idempotency
>>> restriction in the chapter about the greasing process. It's very attractive
>>> to focus on idempotent methods with the restrictive rules and to ignore
>>> everything else.
>>> 
>>> CRS does not do this, but I reckon several vendors have such rules in place in
>>> order to reduce false positives already. The grease process described makes
>>> such a behavior even more attractive. Fighting this from the start will be
>>> important.
>>> 
>>> 
>>> Security considerations
>>> 
>>> I like this section a lot. These considerations are real and even naming them
>>> adds to the quality of the document.
>>> 
>>> 
>>> Appendix A. Bootstrapping the HTTP Grease Process
>>> 
>>> "Broadly speaking, HTTP servers accept these extensions, unless they have a
>>> Web Application Firewall (WAF) installed."
>>> 
>>> This depends on the rule set.
>>> 
>>> Proposed better wording: 
>>> "Broadly speaking, HTTP servers accept these extensions, unless they have a
>>> Web Application Firewall (WAF) installed, which denies them."
>>> 
>>> "WAF vendors" 
>>> 
>>> The CRS project does not necessarily see itself as a "vendor".  I reckon we 
>>> see ourselves as WAF developers. A more inclusive wording would help to
>>> address everybody.
>>> 
>>> "Therefore, a successful greasing strategy needs to include
>>> most or all major clients, and their actions need to be
>>> coordinated."
>>> 
>>> This makes a lot of sense, but if you look at it in a practical way, then it
>>> means that a coordinated greasing effort affects users of different clients in
>>> a similar way. That could mean that users from client A migrate to B and users
>>> of client B migrate to A.  This gives the clients an incentive to minimize the
>>> impact of the greasing effort.  The process described above allows the clients
>>> to make several decisions.  They could be tempted to minimize the greasing for
>>> minimal impact on their users.  That would undermine the effectiveness of the
>>> greasing process.
>>> 
>>> "As a a result, when greasing begins, it will be necessary to have long lead
>>> times between announcement and sending.  Likewise, initial grease values are
>>> more likely to succeed (building confidence and engagement) if they are static
>>> and simple."
>>> 
>>> This is very good.  It may be worthwhile to do a new CRS (point-)release
>>> between announcement and sending.  This would give deployers an incentive to
>>> update quickly, which would be beneficial for our project and maybe others
>>> too.
>>> 
>>> 
>>> 
>>> All in all I like the document, its brevity and the direction it is taking.
>>> I find it a wee bit odd that it describes a process and then also its
>>> first exercise. But I reckon that's no big deal.
>>> 
>>> Best regards,
>>> 
>>> Christian Folini
>>> 
>>> 
>>> -- 
>>> Simplicity is prerequisite for reliability.
>>> -- Edsger W. Dijkstra
>>> 
>>> -- 
>>> Http-grease mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/http-grease
>> 
>> --
>> Mark Nottingham   https://www.mnot.net/

--
Mark Nottingham   https://www.mnot.net/