Re: [Unbearable] Dealing with header injection through reverse proxies

Joseph Salowey <joe@salowey.net> Tue, 18 July 2017 09:27 UTC

Return-Path: <joe@salowey.net>
X-Original-To: unbearable@ietfa.amsl.com
Delivered-To: unbearable@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8F1CC131DC9 for <unbearable@ietfa.amsl.com>; Tue, 18 Jul 2017 02:27:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level:
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=salowey-net.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YAhTiuVLGTDb for <unbearable@ietfa.amsl.com>; Tue, 18 Jul 2017 02:27:31 -0700 (PDT)
Received: from mail-pf0-x231.google.com (mail-pf0-x231.google.com [IPv6:2607:f8b0:400e:c00::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0D9B0131DCE for <unbearable@ietf.org>; Tue, 18 Jul 2017 02:27:31 -0700 (PDT)
Received: by mail-pf0-x231.google.com with SMTP id q85so8759319pfq.1 for <unbearable@ietf.org>; Tue, 18 Jul 2017 02:27:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=salowey-net.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=eardcHx57DyFGnCZvH0PqvCA2VSV9snsk2uxs20wX1A=; b=EPxVmZhbj3xe1RQ7ON9h6uNAAzcTwYfUDk2BZWqK1t3BPs8FFacJBXx8Wum8GLoR4K 0cyIQLDTfqA3qDjQPOd1buWURU6HO0DoVWF8FPyOkN05g8ul0ewfPDLZmUalglTtGG/1 SsdkWv3GOaqaAlFR/ZKa3/QZFetpaGlRbUtobj4hmWUIW7OwtFfYQ4dfVxiBOUdzujNI tEqJETmyM8PxKcCAmDEutUBjIpVd8rx5wiaJKHPm6XwApti7jqy5ZILeJFhUuEaRP+Ao VGQlHo3rFG4phz8/Qk9rI4TBhtdOdka0aXkrHn5UdOzMRni6APbmYGFdqLdhVipW9d/l faAQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=eardcHx57DyFGnCZvH0PqvCA2VSV9snsk2uxs20wX1A=; b=FpmQFATUrquRZvx3+t7GdvqRnf/a/WeX1HmmoLhbof1tGee0BF1XACQ96vpZF6Bnf3 4Lj4FtYXPPs/xXr+Uvg24ajhMSFNcsCfYuVRZkBdpEGBBt/VG5UoJb29+aKYIZn4nUhm gEsSW4AlVTx9u5ejheltB4kl8zjGdypOTYKgKE3x45klmmmH/Kgf5nAY4EAjJoJWuqwj 7wwRC8F/vDU4AekZzLdv2/knLvjetNogOBZ+qtebuEiVw5fHMkuev0jjQ96DGmU7zDCW U+v5HgYPLPCxbA+vLrjI44ZaMNs9YQnoB6Qt9EdsEXbvDpn1KdDUIuLMbQOLjtwsWNmc SPvQ==
X-Gm-Message-State: AIVw111E6+R6UD+rcOwtryV7HpS9GT9JOb+cO5w3XVnfdmuH5jJxfSy3 pDxiBXKVDVDeSul/jZnWEI/ITvypKoHy
X-Received: by 10.84.211.34 with SMTP id b31mr766435pli.230.1500370050629; Tue, 18 Jul 2017 02:27:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.137.2 with HTTP; Tue, 18 Jul 2017 02:27:09 -0700 (PDT)
In-Reply-To: <CA+k3eCTRNebzWO-vf30sp6MZU=yL_rR62-PG8gwn2zW0VZAiaA@mail.gmail.com>
References: <CABcZeBNK4zHCR4V8cRAJBxC0AiVpep8HWoX8Ntnr9ZTZGq8S+A@mail.gmail.com> <CA+k3eCTRNebzWO-vf30sp6MZU=yL_rR62-PG8gwn2zW0VZAiaA@mail.gmail.com>
From: Joseph Salowey <joe@salowey.net>
Date: Tue, 18 Jul 2017 11:27:09 +0200
Message-ID: <CAOgPGoDehDEJLLYV_OR4dTxzbt0vs6qaNADgsV7LVruu+w7-Tw@mail.gmail.com>
To: Brian Campbell <bcampbell@pingidentity.com>
Cc: Eric Rescorla <ekr@rtfm.com>, IETF Tokbind WG <unbearable@ietf.org>, HTTP Working Group <ietf-http-wg@w3.org>
Content-Type: multipart/alternative; boundary="001a114c7a6ce65bfe0554941fbc"
Archived-At: <https://mailarchive.ietf.org/arch/msg/unbearable/G6L48yaQXkVh2d8uwAdDFBGIgrA>
Subject: Re: [Unbearable] Dealing with header injection through reverse proxies
X-BeenThere: unbearable@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "\"This list is for discussion of proposals for doing better than bearer tokens \(e.g. HTTP cookies, OAuth tokens etc.\) for web applications. The specific goal is chartering a WG focused on preventing security token export and replay attacks.\"" <unbearable.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/unbearable>, <mailto:unbearable-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/unbearable/>
List-Post: <mailto:unbearable@ietf.org>
List-Help: <mailto:unbearable-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/unbearable>, <mailto:unbearable-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Jul 2017 09:27:40 -0000

I agree that this is a larger problem than just the token binding headers
and is a problem with proxy generated headers in general.  Also if
sanitization is properly implemented it solves the problem of header
injection, however over the years I have run into many situations where
proxies are mis-configured allowing header injection with sometimes
disastrous consequences.   I'd like to see a more fail-safe approach.

In my experience these deployments usually have the following
characteristics:
1. There is usually configuration required on both proxy and application.
I think it would be possible to coordinate a secret in most cases, but it
would be additional work.
2. TLS is often not deployed between proxy and application, however it
might be OK to require TLS for a secure deployment.
3. H2 is often not deployed, however this may change over time.
4. The proxy often generates multiple headers and it would be good to
protect all of them.

On Tue, Jul 18, 2017 at 10:50 AM, Brian Campbell <bcampbell@pingidentity.com
> wrote:

> I perhaps didn't articulate it well in the meeting yesterday but I'd like
> to reiterate that, in my opinion and I don't think I'm alone, it would be
> very inappropriate for -tokbind-ttrp to define a one-off mechanism for the
> origin server to detect client injection of the headers. The potential
> problem of client header injection is not at all unique to the
> functionality of that draft so the draft shouldn't define a unique
> solution. It would be useful if there were a common standardized mechanism
> for doing detecting/preventing client header injection that the
> -tokbind-ttrp draft could refer to (the generic solution that Ekr mentions
> in his [1] seems preferable precisely because it is generally applicable).
> In the absence of a generic solution existing currently,
> stripping/sanitizing the headers is the de facto means of dealing with the
> situation in practice today, is sufficient when properly implemented, and
> is normatively required by the text in -tokbind-ttrp. It's true that, if
> the reverse proxy is defective/misconfigured, it doesn't fail safe but in
> the context of -tokbind-ttrp that failure mode is far from catastrophic.
> Such a failure loses the protections afforded by token binding, which is
> not ideal, but it is the current state of just about everything on the web
> today so it's not *that* bad.
>
>
>
>
>
>
> On Mon, Jul 17, 2017 at 5:18 PM, Eric Rescorla <ekr@rtfm.com> wrote:
>
>> i folks,
>>
>> We had a discussion today in TOKBIND about handling security-sensitive
>> indications in HTTP headers (this came up in the context of
>> https://tools.ietf.org/html/draft-campbell-tokbind-ttrp-01).  The
>> setting here is that you have a network with a TLS reverse proxy
>> serving the origin server, and the TLS proxy is responsible for doing
>> some security check and telling the server about it. E.g.,
>>
>>     Client                    Proxy                     Server
>>
>>     <--- TLS w/ client auth ---> <----- HTTP with cert --->
>>
>> The client does TLS client authentication with the proxy and then
>> passes the certificate to the back-end server in an injected HTTP
>> header (e.g., X-Client-Certificate). In order for this to be secure,
>> the proxy has to *strip* any security sensitive headers sent by the
>> client. Otherwise, the client could inject their own headers that
>> would appear to come from the proxy.
>>
>> Obviously, this design doesn't fail safe if the proxy fails to
>> strip the headers. One way in which this can happen is if you
>> have a large network of load balancers fronting a server network
>> and you somehow incompletely misconfigure either the servers
>> or the proxies, so that a server which supports this mechanism
>> ends up behind a proxy which does not (and hence does not strip
>> the headers). So, it would be nice to do something better that
>> wasn't too heavyweight, especailly as a general solution.
>>
>> One natural design is simply to have a shared key between the
>> proxy and the server. In that case, it's easy to demonstrate
>> that the header is injected, as in [0][1]
>>
>>    X-Client-Certificate: <key>, <certificate>
>>
>> The obvious objection to this design is that it requires you to
>> establish the shared key, and people were concerned about having to
>> configure it into the proxy. I'm aware of a number of designs here:
>>
>> - Configure it via some sort of remote configuration mechanism.
>> - The proxy could send it to the server on its first request
>>   (might require some trickery on the server).
>> - If you have a TLS channel between Proxy and Server you can
>>   use a TLS exporter.
>> - If you have a H2 connection between the two, you can have the
>>   server provide it in a settings frame.
>> - Use a signature by the proxy using its private key over something
>>   e.g., S(K, "Token binding"). If you use a mostly fixed string,
>>   then the server just needs to verify it once.
>>
>> Do people have other thoughts on this problem? Other good ways to
>> establish the key? Other ideas for how to address it?
>>
>> -Ekr
>>
>>
>> [0] I'd originally suggested HMACing, but that's actually not
>> necessary, though obviously stronger.
>>
>> [1] Note that you could have a generic solution, like a header which
>> listed all the injected headers, but with the same security mechanism.
>>
>>
>> _______________________________________________
>> Unbearable mailing list
>> Unbearable@ietf.org
>> https://www.ietf.org/mailman/listinfo/unbearable
>>
>>
>
> *CONFIDENTIALITY NOTICE: This email may contain confidential and
> privileged material for the sole use of the intended recipient(s). Any
> review, use, distribution or disclosure by others is strictly prohibited.
> If you have received this communication in error, please notify the sender
> immediately by e-mail and delete the message and any file attachments from
> your computer. Thank you.*
> _______________________________________________
> Unbearable mailing list
> Unbearable@ietf.org
> https://www.ietf.org/mailman/listinfo/unbearable
>
>