Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

[Justin Richer <[e-mail]>]

All of these problems can be solved, and I think solved better, by securing the connection between the proxy and the back-end. That way the back end will be able look not only for a specific header, but verify that the header came from the proxy itself. An obscure header name is one way to do that, but it has a lot of issues with it, especially since it’s likely to be selected once during set-up and never changed throughout the lifetime of the deployment. I think there are likely much better solutions here, and they’d address this issue without things getting weird.

And one of the best things about a standard is that you’re still free to completely ignore it if you want to, so people can and will keep following whatever proprietary patterns they want to. But at least a standard mechanism would give us a way to say to newcomers and veterans alike “No really, here’s a way that we all agree works and has these properties”. 

 — Justin

> On Oct 30, 2019, at 3:43 AM, Neil Madden <[e-mail]> wrote:
> 
> Replies below.
> 
>> On 29 Oct 2019, at 19:13, Justin Richer <[e-mail]> wrote:
>> 
>> I would argue that making this standard would actually increase the likelihood of developers getting this right, as now instead of following some copy-pasted recipe for NGINX or Apache that they found on the web, they could turn on a standard setting that would take care of both stripping out incoming headers and injecting the appropriate values. And all of that can be covered in the security considerations with a bunch of normative text on top to make sure inbound headers are stripped. What you’re describing below is clever, but ultimately it’s just a small bit of obscurity more than anything real. 
> 
> Not really. If the header is cryptographically unguessable (eg includes a 128-bit base32-encoded string) then it provides a real security improvement. The header name becomes a bearer token effectively. There are many ways that a reverse proxy can be misconfigured in a way that compromises the security of trusted headers:
> 
> 1. There is the simple misconfiguration where you fail to strip trusted headers from incoming requests. For example in HAProxy this is the difference between set-header and add-header directives (and the add-header docs list passing a client cert to the backend as an example use case [1] despite this being insecure). Naive functional testing will still pass, but fairly basic adversarial tests would catch this. A standard header/config option might help with this simple case. 
> 
> 2. Scoping issues cause rules to not be applied where you expect them to. For example, nginx has multiple levels of scope that you can define header rules (http, server, location). But if you define *any* proxy_set_header directive at the location level (for example) it will ignore *all* such directives at the server or http levels, which can cause header-stripping rules to be silently disabled. 
> 
> 3. More advanced attacks exploit differences in how individual reverse proxies and application servers parse headers to smuggle headers or even whole requests past the RP [2]. 
> 
> Using unguessable header names for transmitting security-critical information between the RP and the app server provides an effective defense in depth against all of these attacks. (They are all forms of confused deputy attack and the unguessable header acts in the same way as an anti-CSRF token to prevent these systematically). 
> 
> (I had thought the random header name pattern was widely known, as I’ve heard it mentioned in conversations several times. But now I come to look for a definitive reference to it I can only find it mentioned in our own docs [3] and connect2id’s [4]). 
> 
> [1]: http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header <http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header>
> [2]: https://portswigger.net/web-security/request-smuggling <https://portswigger.net/web-security/request-smuggling>
> [3]: https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs <https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs>[4]: https://connect2id.com/products/server/docs/guides/tls-proxy <https://connect2id.com/products/server/docs/guides/tls-proxy> 
> 
>> 
>> The way things are today, you’ve got to not only pick a header and figure out its format, but also do the injection protection step yourself. Since all of these are disconnected, there are a lot more places that it could fall over. Even a typo where you throw out incoming “CLIENT_CERT” but inject “CLIENT_CERTS” or something like that would be disastrous. 
> 
> Most load balancers and reverse proxies support some way of setting headers that also strips the incoming header in one directive. Unless the standard gets rid of all the other ways they support to configure this (and removes all the old Stack Overflow recipes) then it won’t necessarily improve things. 
> 
> A standard that provides an explicit field for including a secret (heck, we could even call it an access token :-) that SHOULD be used might provide a significant improvement in security. 
> 
>> 
>> All in all, I am in favor of this being defined in one standard way, in addition to secure communication between proxies and backends being standardized — but this latter bit really seems like a separate problem.
> 
> I agree that that is a separate issue. 
> 
> — Neil
> 
>> 
>>  — Justin
>> 
>>> On Oct 28, 2019, at 12:32 PM, Neil Madden <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> While there are some benefits to standardizing headers for this kind of communication, there are some significant downsides - particularly when using headers to communicate critical security information like certs. It is *very* easy to misconfigure a reverse proxy to not strip Forwarded (or whatever) headers from incoming requests, allowing a client to simply supply a certificate as a header without authenticating the TLS connection with the corresponding private key. One good practice to prevent this is to pick a random and unguessable header name (configurable per installation) to be used for communicating the certificate, rather than using something fixed and standard. That way even if you misconfigure the proxy an attacker still has to try and guess the correct header name.
>>> 
>>> I suppose the same thing could be accomplished by having an extension for including a shared secret (or HMAC tag) in the header to authenticate it.
>>> 
>>> -- Neil
>>> 
>>>> On 28 Oct 2019, at 15:32, Brian Campbell <bcampbell=[e-mail] <mailto:bcampbell=[e-mail]>> wrote:
>>>> 
>>>> I don't think there's anything beyond defining something to carry the client certificate information (including the format and encoding). And it could well be a new RFC7239 parameter. Or it might just be a new HTTP header on its own.  
>>>> 
>>>> On Mon, Oct 28, 2019 at 9:05 AM Rifaat Shekh-Yusef <[e-mail] <mailto:[e-mail]>> wrote:
>>>> Thanks Brian,
>>>> 
>>>> I guess my question is: given RFC7239 and the fact that it is straightforward to secure the channel between the terminating reverse proxy and the backend service in a cluster, is there anything, from a standard perspective, that we need to do beyond defining a new parameter to carry the client certificate information?
>>>> You seem suggest that the answer is yes. If so, can you please elaborate on why is that?
>>>> 
>>>> Regards,
>>>>  Rifaat
>>>> 
>>>> 
>>>> 
>>>> On Mon, Oct 28, 2019 at 8:42 AM Brian Campbell <[e-mail] <mailto:[e-mail]>> wrote:
>>>> 
>>>> 
>>>> On Sat, Oct 26, 2019 at 3:55 PM Rifaat Shekh-Yusef <[e-mail] <mailto:[e-mail]>> wrote:
>>>> 
>>>> On Fri, Oct 25, 2019 at 3:47 PM Brian Campbell <[e-mail] <mailto:[e-mail]>> wrote:
>>>> 
>>>> I did look at RFC7239 when doing that and it could have been made to work but felt the fit wasn't quite right and would have been more cumbersome to use than not.  
>>>> 
>>>> 
>>>> Can you elaborate on this?
>>>> These days, with the zero trust model in mind, there are orchestration tools, e.g. Istio, that easily allows you to establish an MTLS channel between the reverse proxy/load balancer/API GW and the backend servers.
>>>> Why is that not sufficient? 
>>>> Which part is cumbersome?
>>>> 
>>>> What I meant was only that in the course of writing https://tools.ietf.org/html/draft-ietf-tokbind-ttrp-09 <https://tools.ietf.org/html/draft-ietf-tokbind-ttrp-09>, which aims to define HTTP header fields that enable a TLS terminating reverse proxy to convey information to a backend server about the validated Token Binding Message received from a client, it seemed more straightforward and sufficient for the use-case to use new HTTP headers to carry the information rather than to use new fields in the Forwarded header framework from RFC7239.    
>>>>  
>>>> 
>>>> CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited.  If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. Thank you.
>>>> 
>>>> CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited..  If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. Thank you._______________________________________________
>>>> OAuth mailing list
>>>> [e-mail] <mailto:[e-mail]>
>>>> https://www.ietf.org/mailman/listinfo/oauth <https://www.ietf.org/mailman/listinfo/oauth>
>>> _______________________________________________
>>> OAuth mailing list
>>> [e-mail] <mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/oauth
>>