Re: [OAUTH-WG] [UNVERIFIED SENDER] Re: client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

Neil Madden <> Fri, 01 November 2019 09:06 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 15B3B120152 for <>; Fri, 1 Nov 2019 02:06:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.997
X-Spam-Status: No, score=-1.997 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id JD_J0Jkjxz_c for <>; Fri, 1 Nov 2019 02:06:26 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::333]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 8AEC412012E for <>; Fri, 1 Nov 2019 02:06:25 -0700 (PDT)
Received: by with SMTP id 8so1405836wmo.0 for <>; Fri, 01 Nov 2019 02:06:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=google; h=content-transfer-encoding:from:mime-version:subject:message-id:date :cc:to; bh=SV34MieBGahjsaNFfJQShsOXpym3D3y3v9kqoHMBIYE=; b=W2O58ecG2FIeset/NmmWnA/0khLcJexK1P1G1tlGOcWjrgIfqpONPM+u63Cz4P7xeF loasr1YrbhF0OIeINOFgY4+6sueIPo9eF9g8nPRW6Uul7N6isFiiKMWRRJBenvEFrDNK dX3U5L6FsjwKOPqK9i1CFPktO4M641YlIdCqU=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:message-id:date:cc:to; bh=SV34MieBGahjsaNFfJQShsOXpym3D3y3v9kqoHMBIYE=; b=ff2TCfwiYU/HVzaQMuyc5o0IVlWPYovzvcEKE6lblwJY/rbDuYW+g+ChJGQ4bv6UDp V+M9dJaGjI8fpOO53SvWBh0HYfSWL/hI6Aegpa2LN/xzwQDlQLFW1Lpld82u6PP2q8kF Ntn0vNXr3Y0krBXsV/gOwcaPqxqxmI2dxOAIDj1dNcZ2X/CFNcjd0bWLN999aPwAfW6b X8bfN5hapZMUdvYxVKRXFtmK0+HTvuaeM/be0cA6xDtTRGydtyEh6+PXLs1umTKg3WyO dTh1lXEmqESWLc65Zj2diWuhvBrCQnOHzcEkRFeeB3aVoSmjbBHIo22IwmMlvC/dT5Bz HSMg==
X-Gm-Message-State: APjAAAXqZlcx2HVO8ugbVxdwiLpQuSlDdP/aBlEyppuZexDVaDXA8kgW 4bjgpjGlpAP4YOxlCxnFlVpW1ukNIG5HHFfxp029hDrv/tMHqAeETECCBHfePq0b7sWiuOEraoq taHqbsxAr8tywjmymox48mdQ/G4Z+1zFEVyCfsyiQUE7FsrnHWjgbvDvTLQjCiTw=
X-Google-Smtp-Source: APXvYqyLbPQ+TUc4jn+2qAAYFCv7SGrMK8m+srUPBQ2rp5HBTf1eaLINzddrUbe86f6hiqs+JavFvw==
X-Received: by 2002:a1c:6a1a:: with SMTP id f26mr2325722wmc.19.1572599183297; Fri, 01 Nov 2019 02:06:23 -0700 (PDT)
Received: from ?IPv6:2a01:4c8:f:b76f:8df3:da5f:6b92:be99? ([2a01:4c8:f:b76f:8df3:da5f:6b92:be99]) by with ESMTPSA id q14sm8914745wre.27.2019. (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 01 Nov 2019 02:06:22 -0700 (PDT)
Content-Type: multipart/alternative; boundary="Apple-Mail-36F29124-5779-4973-AA0D-E72B19D5EDD4"
Content-Transfer-Encoding: 7bit
From: Neil Madden <>
Mime-Version: 1.0 (1.0)
Message-Id: <>
Date: Fri, 01 Nov 2019 09:06:20 +0000
Cc: "" <>
To: "Richard Backman, Annabelle" <>
X-Mailer: iPhone Mail (17A878)
Archived-At: <>
Subject: Re: [OAUTH-WG] [UNVERIFIED SENDER] Re: client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: OAUTH WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 01 Nov 2019 09:06:28 -0000

>> On 31 Oct 2019, at 21:09, Richard Backman, Annabelle <> wrote:
> The comparison the bearer tokens is illustrative of the problems I and others are pointing out:
> Bearer tokens are embedded in the value of the header, not the header itself, which partially alleviates the concern I raised regarding request signing algorithms.
> Bearer tokens are typically relatively short-lived, providing some mitigation against exfiltration through logs.
> Bearer tokens are typically dynamically generated, and are therefore less likely to be embedded in source code or config files in a project’s repository.
> Bearer tokens are called tokens, and are presented as secrets and are always expected to be treated as secrets.
> The random header name is effectively an infinite-lifetime, statically defined bearer token presented in a way that does not at all make clear and obvious that it is a secret that must be protected, and in fact makes it more likely that it will be revealed, rendering it useless. And like all bearer tokens, even under ideal conditions it by definition CANNOT BE USED TO AUTHENTICATE THE SENDER.

Are you saying bearer credentials cannot be used for authentication? Passwords? Session cookies? API keys?

You can change the header name periodically if you really think it might leak.

> There is a certain amount of irony in the idea of the security of a Mutual TLS deployment ultimately coming down to a bearer-token header name passed between the reverse proxy and the protected service. 😂

Well if you don’t do this then in the situations where it would come down to a bearer token, it currently comes down to absolutely nothing at all.

Let’s have a look at some concrete examples, as they are rather easy to find. 

In 2014, Tomcat found and fixed CVE-2014-0099 that was an integer overflow bug that allowed arbitrary request smuggling when running behind a reverse proxy, bypassing all request header sanitisation in the RP. If your app was trusting an X-SSL-Cert header for authentication then this is likely to lead to complete compromise.

Here’s an example of some people who spent hours after a DoS attack before they realised all the IP addresses in their server logs were spoofed because HAProxy hadn’t been configured to strip X-Forwarded-For headers that the backend was trusting.

Here’s another one with somebody wondering why their logs contain private space IP addresses:

Common theme: apps often blindly trust standard header names by default, while reverse proxies/load balancers commonly forward them by default. 

Here’s a security researcher writing about how to spoof an SSL client certificate header and listing the commonly used header names for several reverse proxies, along with simple ways to bypass header sanitisation:

Here’s an old critical vulnerability in Shibboleth that led to arbitrary spoofing of security attributes due to ambiguous normalization of header names after filtering:

Here’s an example of people misconfiguring Envoy’s support for client certificate forwarding and then discovering it can be spoofed. 

Here’s an example of an app being misconfigured when parsing X-Forwarded-For headers behind an RP leading to a bypass of API key authentication:

If you search around of google or stack overflow you can easily find dozens more examples of people being bitten by misconfigured RPs and apps trusting headers. There is a good reason why Security Misconfiguration is in the OWASP top 10. 

Adding a random string to a header name means that in the majority of cases these all-too-common failures don’t immediately lead to a catastrophic compromise of your app. Sure, its not a perfect solution, but it usually stops a common vulnerability being immediately exploitable. Wouldn’t you rather that an attacker has to go to a little more trouble than trying a list of a dozen widely used header names in this situation?

It is not helpful to tell people not to make such a simple and essentially cost-free tweak to their configuration because in some world they might enable trace-level logging of all request headers in a production environment, or their network engineers might post packet captures in powerpoint presentations, or they might simultaneously be using headers to communicate security information to the backend while allowing untrusted clients on the same network to sniff network traffic in the clear.

The alternative at the moment is to do nothing at all, which we know leads to potential vulnerabilities because there is ample evidence in the public record. Stack overflow is a catalogue of people confused about configuring reverse proxy header forwarding. And yet This One Weird Trick Could Save You From Reverse Proxy Trusted Header Misconfiguration Vulnerabilities!

> > if you forget to validate the signature *nothing fails*
> Replace the HMAC with encryption and you solve that problem, as it forces the service to decrypt the value using the correct key in order to access the client certificate data. Whether or not that’s worth doing is something we can debate in the context of an actual proposal.

Sure, and if the final proposal for a standard header comes with optional/recommended support for authenticated encryption of values (and reverse proxy vendors actually implement it) then I’ll be happy enough. And when that finally reaches a good level of adoption I’d advise people to use it. Just don’t make it a JWT with a choice of a dozen algorithms. 

— Neil