Re: [Masque] Forwarded Mode infinite loops

[Alex Chernyakhovsky <[e-mail]>]

I chatted with David about this earlier today to better understand the
problem and proposed solution, and one of the conclusions I left the
discussion with is that Ian's point about decrementing the TTL is
Necessary, but possibly not Sufficient, mitigation. It does mean that a
straightforward software implementation of the proxy doing recvmsg();
sendmsg(); is insecure, and you need to do the appropriate setsockopts to
IP_RECVTTL / set IP_TTL on each outgoing packet to turn the loop from
infinite to finite.

However, the thing that I find ideologically unsatisfying is that this does
nothing to mitigate the case where the Target is turned into a
confused-deputy to send unwanted traffic to a given client, because there's
no way for the final egressing proxy (P1) to be sure the client used a
chosen connection ID that does not conflict/overlap. It seems like we need
to consider a QUIC-level feature to perhaps make this possible/mandatory to
prevent this class of attack. I don't think that's possible to do with QUIC
as formulated today and would probably require some changes.

Sincerely,
-Alex

On Tue, Oct 24, 2023 at 3:37 PM Martin Duke <[e-mail]> wrote:

> That is pretty much a photo of my whiteboard when crafting the
> original email!
>
> ISTM the design has to give up on one of three principles and/or
> operational degrees of freedom:
> (1) Receiver picks its connection ID.
> (2) Proxies don't need to use server preferred address to use this feature
> (3) Proxies don't have to open a new target-facing UDP socket for each
> CONNECT (unlike CONNECT-UDP!)
>
> IIUC you are proposing eliminating (2), which to me sounds like the
> biggest constraint to add. But you guys are the authors, so do what you
> think is best.
>
> I also have a sense that (2) is a point solution to the only attack
> someone has managed to assemble, and that other problems of this nature are
> out there. But that's just conjecture at this point.
>
> Martin
>
> On Tue, Oct 24, 2023 at 11:52 AM David Schinazi <[e-mail]>
> wrote:
>
>> re: Martin and Tommy, thanks for the clarification about ingress vs
>> egress - that's super helpful. My whiteboard was starting to look like this:
>>
>> https://i.kym-cdn.com/entries/icons/original/000/022/524/pepe_silvia_meme_banner.jpg
>>
>
> At a more fundamental level, the issue stems from the fact that we're
>> breaking one of the rules of QUIC. In QUIC, the general mindset is that the
>> receiver of a Destination Connection ID is the one that picks this CID. The
>> draft violates that in a few ways:
>> * The Client Connection ID is picked by the client even though it's
>> received by the proxy egress. This one's OK because the proxy gets to
>> accept or reject the proposed CID.
>> * The Target Connection ID is picked by the target, but communicated to
>> the proxy by the client, so the client could lie. I haven't been able to
>> mount an attack using that fact though.
>> * The Virtual Client CID is picked by the client. That's perfectly fine
>> in a single instance of this draft, but once you chain two of these
>> (C-P1-P2-T) then the client gets to pick the Virtual Client CID that P2
>> sends to P1. This is the one that causes the attack here, Client is picking
>> a CID that P1 receives.
>>
>> I'm not sure that having the proxy pick the Virtual Client CID is a good
>> fix here because that also violates the QUIC rule of receiver picks CIDs
>> and could lead to other issues where the proxy is the attacker.
>>
>
>
>>
>> A mitigation we've discussed is the fact that the proxy could look at the
>> target ingress IP when routing. That breaks when multiple proxies share an
>> ingress IP. Perhaps we can prevent that by requiring something similar to
>> server preferred address where the proxy can only use forwarded mode on an
>> ingress IP that is unique to them?
>>
>
>
>>
>> David
>>
>> On Tue, Oct 24, 2023 at 9:57 AM Martin Duke <[e-mail]>
>> wrote:
>>
>>> That's a good question!
>>>
>>> IIUC, CONNECT-IP does decrement the TTL, but CONNECT-UDP does not
>>> because it is not preserving the IP header (The CONNECT-IP draft mentions
>>> TTL and CONNECT-UDP does not). The quic-proxy draft doesn't say
>>> anything about this, so I imagine that rule still applies.
>>>
>>> On Tue, Oct 24, 2023 at 6:43 AM Ian Swett <[e-mail]> wrote:
>>>
>>>> I assume each hop would decrement the TTL?  If so, wouldn't this be a
>>>> loop, but not infinite?
>>>>
>>>> Ian
>>>>
>>>> On Mon, Oct 23, 2023, 11:01 PM Tommy Pauly <tpauly=
>>>> [e-mail]> wrote:
>>>>
>>>>> It was useful for me to draw this out in (unnecessary) detail to show
>>>>> how this only needs the ingress IP to be shared, not the egress IP.
>>>>>
>>>>> To help go through this, let me fill in the example you gave with more
>>>>> details about IPs and ports — since the forwarding mappings depend on these
>>>>> “sockets”.
>>>>>
>>>>> Let’s start with a case where the addresses aren’t being shared.
>>>>>
>>>>> *C *has address 1111::1
>>>>> *P1* has ingress address 2222::2, and egress address 3333::3
>>>>> *P2* has ingress address 4444::4, and egress address 5555::5
>>>>> *P3* has ingress address 6666::6, and egress address 7777::7
>>>>> *T* has ingress address 8888::8
>>>>>
>>>>> *C *is set up to receive CID4 on 1111::1 port 10000 from 2222::2 port
>>>>> 443
>>>>> *P1 *is set up to receive CID3 on 3333::3 port 20000 from 4444::4
>>>>> port 443, and forward it as CID4 to 1111::1 port 10000 from 2222::2
>>>>> port 443
>>>>> *P2 *is set up to receive CID2 on 5555::5 port 30000 from 2222::2
>>>>> port 443, and forward it as CID3 to 3333::3 port 20000 from 4444::4
>>>>> port 443
>>>>> *P1 *is set up to receive CID1 on 3333::3 port 40000 from 6666::6
>>>>> port 443, and forward it as CID2 to 5555::5 port 30000 from 2222::2
>>>>> port 443
>>>>> *P3 *is set up to receive CID0 on 7777::7 port 50000 from 8888::8
>>>>> port 443, and forward it as CID1 to 3333::3 port 40000 from 6666::6
>>>>> port 443
>>>>> *T *sends using CID0 to 7777::7 port 50000 from 8888::8 port 443
>>>>>
>>>>> In this case, since there is no address sharing, if the client tells
>>>>> *P2* to instead map from (CID2 on 5555::5 port 30000 from 2222::2
>>>>> port 443) to (CID1 to 3333::3 port 20000 from 4444::4 port 443), the
>>>>> connection will break, but there will be no loop, since *P1* won’t
>>>>> allow receiving CID1 on that socket.
>>>>>
>>>>>
>>>>> Then, let’s explore the case of shared addresses. If we only have P2
>>>>> and P3 share an ingress VIP, and P1 reuses sockets on egress, we have the
>>>>> following:
>>>>>
>>>>> *C *has address 1111::1
>>>>> *P1* has ingress address 2222::2, and egress address 3333::3
>>>>> *P2* has ingress address 4444::4, and egress address 5555::5
>>>>> *P3* has ingress address 4444::4, and egress address 7777::7
>>>>> *T* has ingress address 8888::8
>>>>>
>>>>> *C *is set up to receive CID4 on 1111::1 port 10000 from 2222::2 port
>>>>> 443
>>>>> *P1 *is set up to receive CID3 on 3333::3 port 20000 from 4444::4
>>>>> port 443, and forward it as CID4 to 1111::1 port 10000 from 2222::2
>>>>> port 443
>>>>> *P2 *is set up to receive CID2 on 5555::5 port 30000 from 2222::2
>>>>> port 443, and forward it as CID3 to 3333::3 port 20000 from 4444::4
>>>>> port 443
>>>>> *P1 *is set up to receive CID1 on 3333::3 port 20000 from 4444::4
>>>>> port 443, and forward it as CID2 to 5555::5 port 30000 from 2222::2
>>>>> port 443
>>>>> *P3 *is set up to receive CID0 on 7777::7 port 50000 from 8888::8
>>>>> port 443, and forward it as CID1 to 3333::3 port 20000 from 4444::4
>>>>> port 443
>>>>> *T *sends using CID0 to 7777::7 port 50000 from 8888::8 port 443
>>>>>
>>>>> Then, if the client tells *P2* to instead map from (CID2 on 5555::5
>>>>> port 30000 from 2222::2 port 443) to (CID1 to 3333::3 port 20000 from
>>>>> 4444::4 port 443), the loop occurs because there are two mappings for
>>>>> receiving on *P1 *that differ only in CID.
>>>>>
>>>>> Tommy
>>>>>
>>>>> On Oct 23, 2023, at 7:08 PM, Martin Duke <[e-mail]>
>>>>> wrote:
>>>>>
>>>>> The reason I put in the point about VIPs is that P1 cannot distinguish
>>>>> between a packet from P2 and P3 if they share an ingress VIP
>>>>>
>>>>> If they're from different IP addresses it's sufficient for P1 to check
>>>>> the source IP of packets arriving from the target
>>>>>
>>>>> Does that answer your question?
>>>>>
>>>>>
>>>>> On Mon, Oct 23, 2023, 18:34 David Schinazi <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> I don't understand how the fact that P2 and P3 share a VIP impacts the
>>>>>> attack. Thinking about this some more, maybe the issue is that we mean
>>>>>> different things when we say VIP? In my mind:
>>>>>>
>>>>>> * a "VIP" is an (IP address, UDP port) tuple. The V stands for
>>>>>> virtual in the
>>>>>>   sense that two separate servers can both have the same IP due to
>>>>>> anycast.
>>>>>>
>>>>>> * an "ingress VIP" is the VIP that the client sends packets to. In
>>>>>> the document
>>>>>>   this is called a "client-facing socket".
>>>>>>
>>>>>> * an "egress VIP" is the VIP that the target sends packets to. In the
>>>>>> document
>>>>>>   this is called a "target-facing socket".
>>>>>>
>>>>>> Ingress VIPs are very common in practice. Many CDNs will use anycast
>>>>>> and
>>>>>> have an ingress VIP that represents a fleet of frontends. Egress VIPs
>>>>>> however,
>>>>>> don't make any sense to me, because you'd never want anycast on
>>>>>> egress IPs.
>>>>>>
>>>>>> Could you help reframe the P2/P3 shared VIP point in terms of ingress
>>>>>> and
>>>>>> egress VIPs please?
>>>>>>
>>>>>> Thanks,
>>>>>> David
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 20, 2023 at 3:00 PM Martin Duke <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>> As I continue to think about this, I'm not sure that
>>>>>>> authentication even solves the problem. The client could authenticate with
>>>>>>> each client, sure, but to P1, P2 and P3 are essentially targets. Targets,
>>>>>>> obviously will not do anything special just because this is a connection
>>>>>>> over MASQUE, so there cannot be a special mechanism for P1 to verify
>>>>>>> authenticity of packets from the target, unless it has the target-client
>>>>>>> session keys.
>>>>>>>
>>>>>>> On Fri, Oct 20, 2023 at 12:14 PM Martin Duke <
>>>>>>> [e-mail]> wrote:
>>>>>>>
>>>>>>>> I should add that another way to thwart this attack is to require
>>>>>>>> the proxy to open a new target-facing socket for every client request. Then
>>>>>>>> the packet incoming to P1 would be disambiguated by the destination
>>>>>>>> IP/port. Not opening a new socket is currently allowed in the spec, which
>>>>>>>> IIUC is a change from CONNECT-UDP.
>>>>>>>>
>>>>>>>> So the proxy vulnerability occurs if
>>>>>>>> (1) P2/P3 share a VIP,
>>>>>>>> (2) P1 is not load balanced, so that the attacker can directly
>>>>>>>> target it, and
>>>>>>>> (3) P1 is not opening unique target-facing sockets for each
>>>>>>>> request, even though the target IP/port are the same.
>>>>>>>>
>>>>>>>> On Fri, Oct 20, 2023 at 11:56 AM Martin Duke <
>>>>>>>> [e-mail]> wrote:
>>>>>>>>
>>>>>>>>> Forwarded packets from the client SHOULD count. The draft is
>>>>>>>>> silent about forwarded packets from the target.
>>>>>>>>>
>>>>>>>>> But fair enough, the client needs to do just enough to keep it
>>>>>>>>> alive.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 20, 2023 at 11:15 AM Ben Schwartz <[e-mail]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I'm not sure the client can actually "go away".  If the HTTP
>>>>>>>>>> connection ends, the forwarding sessions will also end, so this can be
>>>>>>>>>> addressed by setting a QUIC max_idle_timeout.
>>>>>>>>>>
>>>>>>>>>> This assumes that forwarded packets don't count toward the idle
>>>>>>>>>> timeout, which they shouldn't.  We might need to say that in the draft.
>>>>>>>>>>
>>>>>>>>>> --Ben
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Masque <[e-mail]> on behalf of Martin
>>>>>>>>>> Duke <[e-mail]>
>>>>>>>>>> *Sent:* Friday, October 20, 2023 1:47 PM
>>>>>>>>>> *To:* Ben Schwartz <[e-mail]>
>>>>>>>>>> *Cc:* MASQUE <[e-mail]>
>>>>>>>>>> *Subject:* Re: [Masque] Forwarded Mode infinite loops
>>>>>>>>>>
>>>>>>>>>> Ben, Replies inline. On Fri, Oct 20, 2023 at 9: 53 AM Ben
>>>>>>>>>> Schwartz <bemasc@ meta. com> wrote: Loops! This is a fun attack.
>>>>>>>>>> It seems like there are probably lots of loopy configurations if you have
>>>>>>>>>> access to two or more proxies. Most of the
>>>>>>>>>> Ben,
>>>>>>>>>>
>>>>>>>>>> Replies inline.
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 20, 2023 at 9:53 AM Ben Schwartz <[e-mail]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Loops!  This is a fun attack.  It seems like there are probably
>>>>>>>>>> lots of loopy configurations if you have access to two or more proxies.
>>>>>>>>>>
>>>>>>>>>> Most of the encryption discussion for forwarded mode has focused
>>>>>>>>>> on unauthenticated encryption, which wouldn't help here.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have not seen a written proposal in this space, but if the
>>>>>>>>>> encryption scheme has no authentication properties it doesn't mitigate
>>>>>>>>>> pretty basic DoS attacks. Perhaps this attack creates a new design
>>>>>>>>>> requirement for encryption?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Changing who chooses the Connection IDs could help, but
>>>>>>>>>> proxy-chosen CIDs are not necessarily high entropy, so there's no clear
>>>>>>>>>> guarantee.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> IIUC your contention is that this arises when P3 and P2 both
>>>>>>>>>> happen to select CID1 as the virtual connection ID. A well-intentioned
>>>>>>>>>> client would therefore register CID1 to P1 twice, and P1 would reject it as
>>>>>>>>>> a duplicate in one of those instances, avoiding a loop.
>>>>>>>>>>
>>>>>>>>>> However, a malicious client could note the conflict and just
>>>>>>>>>> register CID3 to P1 instead, creating the same attack.
>>>>>>>>>>
>>>>>>>>>> I do think that if we try to fix this with proxy-chooses, there
>>>>>>>>>> need to be some minimum size limits to avoid this kind of thing. If CIDs
>>>>>>>>>> are 64 bits (or even 32!) I feel pretty good about this not being an
>>>>>>>>>> exploitable attack.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ultimately, I'm not sure this matters.  A hostile client can
>>>>>>>>>> impose pretty much arbitrarily amplified load on the forwarder anyway, e.g.
>>>>>>>>>> by downloading a large file and forging ACKs to dropped packets so that
>>>>>>>>>> congestion control ramps up to infinity.  Forwarders will need defenses to
>>>>>>>>>> rate-limit or cap abusive clients.  That said, it's probably worth fixing
>>>>>>>>>> if there's an easy solution.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I agree there are other attacks -- in fact, I have other
>>>>>>>>>> amplification attacks in mind, which will write about another day.
>>>>>>>>>>
>>>>>>>>>> What is particularly pernicious about this attack is that a
>>>>>>>>>> single attacker packet results in infinity packets in the proxy path. This
>>>>>>>>>> flow can be "rate limited" but the client can go away and this persists
>>>>>>>>>> forever.
>>>>>>>>>>
>>>>>>>>>> Buit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --Ben
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Masque <[e-mail]> on behalf of Martin
>>>>>>>>>> Duke <[e-mail]>
>>>>>>>>>> *Sent:* Wednesday, October 18, 2023 5:25 PM
>>>>>>>>>> *To:* MASQUE <[e-mail]>
>>>>>>>>>> *Subject:* [Masque] Forwarded Mode infinite loops
>>>>>>>>>>
>>>>>>>>>> Hello MASQUE, (with no hats on) I was playing with some annoying
>>>>>>>>>> but manageable amplification attacks on CONNECT-UDP forwarding mode. Those
>>>>>>>>>> will have to wait for another day, because the current draft has a fatal
>>>>>>>>>> vulnerability, though one that
>>>>>>>>>> Hello MASQUE,
>>>>>>>>>>
>>>>>>>>>> (with no hats on)
>>>>>>>>>>
>>>>>>>>>> I was playing with some annoying but manageable amplification
>>>>>>>>>> attacks on CONNECT-UDP forwarding mode. Those will have to wait for another
>>>>>>>>>> day, because the current draft has a fatal vulnerability, though one that
>>>>>>>>>> we can fix pretty easily.
>>>>>>>>>>
>>>>>>>>>> If I've gotten something incorrect, please say so! It wouldn't be
>>>>>>>>>> the first time.
>>>>>>>>>>
>>>>>>>>>> Recall that for forwarded mode, clients notify the proxy of the
>>>>>>>>>> connection ID they are advertising to the target, and in the same
>>>>>>>>>> capsule propose a "virtual connection ID" for use on the proxy-client path.
>>>>>>>>>> (IIUC, this virtual CID is to prevent association of the packets on either
>>>>>>>>>> end of the proxy).
>>>>>>>>>>
>>>>>>>>>> I reiterate that we are talking about the CLIENT connection ID
>>>>>>>>>> and its virtual counterparts, not the target.
>>>>>>>>>>
>>>>>>>>>> Consider the following very stupid topology that a client C could
>>>>>>>>>> construct to target T via various proxies. Crucially, P2 and P3 are from
>>>>>>>>>> the same provider and share a VIP.* This wrinkle eliminates some really
>>>>>>>>>> simple mitigations for what I'm about to describe.
>>>>>>>>>>
>>>>>>>>>> C -> P1 -> P2 -> P1 -> P3 ->  T
>>>>>>>>>>
>>>>>>>>>> An otherwise well-intentioned client would advertise client
>>>>>>>>>> Connection ID CID0 to T, and via REGISTER_CLIENT_CID capsules might set up
>>>>>>>>>> the rest like so:
>>>>>>>>>>
>>>>>>>>>> Nodes        C     P1     P2    P1    P3     T
>>>>>>>>>> CID on wire    CID4   CID3  CID2  CID1   CID0
>>>>>>>>>>
>>>>>>>>>> So far, so good. A packet that leaves T will be destined for
>>>>>>>>>> CID0. The destination CID will be rewritten in turn, arriving at C with
>>>>>>>>>> CID4.
>>>>>>>>>>
>>>>>>>>>> However, a malicious client can register CIDs with the proxies as
>>>>>>>>>> follows:
>>>>>>>>>> Nodes        C     P1            P2      P1      P3     T
>>>>>>>>>> CID on wire    CID4   CID3 / CID1    CID2    CID1   CID0
>>>>>>>>>>
>>>>>>>>>> The only change is that P2 is "incorrectly" configured by the
>>>>>>>>>> client to rewrite incoming CID2 as CID1. As far as P2 is concerned,
>>>>>>>>>> everything is seems legit unless it is sharing enormous amounts of CID
>>>>>>>>>> state with its partner P3. Everything else is unchanged.
>>>>>>>>>>
>>>>>>>>>> Thus, a packet leaving T with CID0 will be rewritten to CID1 and
>>>>>>>>>> routed to P1;  P1 will then route it to P2 with CID2. Then that gets routed
>>>>>>>>>> back to P1 with CID1, and the P1-P2-P1 loop continues to infinity. The
>>>>>>>>>> attacker can put as many proxy hops as it likes in that chain.
>>>>>>>>>>
>>>>>>>>>> There are a few pretty simple things that can break this doom
>>>>>>>>>> loop.
>>>>>>>>>> - Do encryption at each MASQUE hop, as previously proposed. This
>>>>>>>>>> verifies the previous hop of the packet and drops those where the CID does
>>>>>>>>>> not have the correct origin.
>>>>>>>>>> - Have the proxy provide the virtual CID instead of the client.
>>>>>>>>>> Client control is the root of these problems.
>>>>>>>>>> - Eliminate virtual client CIDs and just take the L on
>>>>>>>>>> linkability in that case
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>> * Interestingly, this attack requires at least one proxy in the
>>>>>>>>>> path (P1 in this case) to not be load balanced or have an easily
>>>>>>>>>> manipulated load balancer, so that two QUIC connection attempts can with
>>>>>>>>>> high probability go to the same device.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>> Masque mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/masque
>>>>>>>
>>>>>> --
>>>>> Masque mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/masque
>>>>>
>>>>>
>>>>> --
>>>>> Masque mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/masque
>>>>>
>>>> --
> Masque mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/masque
>