Re: [Masque] Forwarded Mode infinite loops

[Tommy Pauly <[e-mail]>]

It was useful for me to draw this out in (unnecessary) detail to show how this only needs the ingress IP to be shared, not the egress IP.

To help go through this, let me fill in the example you gave with more details about IPs and ports — since the forwarding mappings depend on these “sockets”.

Let’s start with a case where the addresses aren’t being shared.

C has address 1111::1
P1 has ingress address 2222::2, and egress address 3333::3
P2 has ingress address 4444::4, and egress address 5555::5
P3 has ingress address 6666::6, and egress address 7777::7
T has ingress address 8888::8

C is set up to receive CID4 on 1111::1 port 10000 from 2222::2 port 443
P1 is set up to receive CID3 on 3333::3 port 20000 from 4444::4 port 443, and forward it as CID4 to 1111::1 port 10000 from 2222::2 port 443
P2 is set up to receive CID2 on 5555::5 port 30000 from 2222::2 port 443, and forward it as CID3 to 3333::3 port 20000 from 4444::4 port 443
P1 is set up to receive CID1 on 3333::3 port 40000 from 6666::6 port 443, and forward it as CID2 to 5555::5 port 30000 from 2222::2 port 443
P3 is set up to receive CID0 on 7777::7 port 50000 from 8888::8 port 443, and forward it as CID1 to 3333::3 port 40000 from 6666::6 port 443
T sends using CID0 to 7777::7 port 50000 from 8888::8 port 443

In this case, since there is no address sharing, if the client tells P2 to instead map from (CID2 on 5555::5 port 30000 from 2222::2 port 443) to (CID1 to 3333::3 port 20000 from 4444::4 port 443), the connection will break, but there will be no loop, since P1 won’t allow receiving CID1 on that socket.


Then, let’s explore the case of shared addresses. If we only have P2 and P3 share an ingress VIP, and P1 reuses sockets on egress, we have the following:

C has address 1111::1
P1 has ingress address 2222::2, and egress address 3333::3
P2 has ingress address 4444::4, and egress address 5555::5
P3 has ingress address 4444::4, and egress address 7777::7
T has ingress address 8888::8

C is set up to receive CID4 on 1111::1 port 10000 from 2222::2 port 443
P1 is set up to receive CID3 on 3333::3 port 20000 from 4444::4 port 443, and forward it as CID4 to 1111::1 port 10000 from 2222::2 port 443
P2 is set up to receive CID2 on 5555::5 port 30000 from 2222::2 port 443, and forward it as CID3 to 3333::3 port 20000 from 4444::4 port 443
P1 is set up to receive CID1 on 3333::3 port 20000 from 4444::4 port 443, and forward it as CID2 to 5555::5 port 30000 from 2222::2 port 443
P3 is set up to receive CID0 on 7777::7 port 50000 from 8888::8 port 443, and forward it as CID1 to 3333::3 port 20000 from 4444::4 port 443
T sends using CID0 to 7777::7 port 50000 from 8888::8 port 443

Then, if the client tells P2 to instead map from (CID2 on 5555::5 port 30000 from 2222::2 port 443) to (CID1 to 3333::3 port 20000 from 4444::4 port 443), the loop occurs because there are two mappings for receiving on P1 that differ only in CID.

Tommy

> On Oct 23, 2023, at 7:08 PM, Martin Duke <[e-mail]> wrote:
> 
> The reason I put in the point about VIPs is that P1 cannot distinguish between a packet from P2 and P3 if they share an ingress VIP 
> 
> If they're from different IP addresses it's sufficient for P1 to check the source IP of packets arriving from the target 
> 
> Does that answer your question?
> 
> 
> On Mon, Oct 23, 2023, 18:34 David Schinazi <[e-mail] <mailto:[e-mail]>> wrote:
>> Hi Martin,
>> 
>> I don't understand how the fact that P2 and P3 share a VIP impacts the
>> attack. Thinking about this some more, maybe the issue is that we mean
>> different things when we say VIP? In my mind:
>> 
>> * a "VIP" is an (IP address, UDP port) tuple. The V stands for virtual in the
>>   sense that two separate servers can both have the same IP due to anycast.
>> 
>> * an "ingress VIP" is the VIP that the client sends packets to. In the document
>>   this is called a "client-facing socket".
>> 
>> * an "egress VIP" is the VIP that the target sends packets to. In the document
>>   this is called a "target-facing socket".
>> 
>> Ingress VIPs are very common in practice. Many CDNs will use anycast and
>> have an ingress VIP that represents a fleet of frontends. Egress VIPs however,
>> don't make any sense to me, because you'd never want anycast on egress IPs.
>> 
>> Could you help reframe the P2/P3 shared VIP point in terms of ingress and
>> egress VIPs please?
>> 
>> Thanks,
>> David
>> 
>> 
>> On Fri, Oct 20, 2023 at 3:00 PM Martin Duke <[e-mail] <mailto:[e-mail]>> wrote:
>>> As I continue to think about this, I'm not sure that authentication even solves the problem. The client could authenticate with each client, sure, but to P1, P2 and P3 are essentially targets. Targets, obviously will not do anything special just because this is a connection over MASQUE, so there cannot be a special mechanism for P1 to verify authenticity of packets from the target, unless it has the target-client session keys.
>>> 
>>> On Fri, Oct 20, 2023 at 12:14 PM Martin Duke <[e-mail] <mailto:[e-mail]>> wrote:
>>>> I should add that another way to thwart this attack is to require the proxy to open a new target-facing socket for every client request. Then the packet incoming to P1 would be disambiguated by the destination IP/port. Not opening a new socket is currently allowed in the spec, which IIUC is a change from CONNECT-UDP.
>>>> 
>>>> So the proxy vulnerability occurs if 
>>>> (1) P2/P3 share a VIP,
>>>> (2) P1 is not load balanced, so that the attacker can directly target it, and
>>>> (3) P1 is not opening unique target-facing sockets for each request, even though the target IP/port are the same.
>>>> 
>>>> On Fri, Oct 20, 2023 at 11:56 AM Martin Duke <[e-mail] <mailto:[e-mail]>> wrote:
>>>>> Forwarded packets from the client SHOULD count. The draft is silent about forwarded packets from the target.
>>>>> 
>>>>> But fair enough, the client needs to do just enough to keep it alive.
>>>>> 
>>>>> On Fri, Oct 20, 2023 at 11:15 AM Ben Schwartz <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>> I'm not sure the client can actually "go away".  If the HTTP connection ends, the forwarding sessions will also end, so this can be addressed by setting a QUIC max_idle_timeout.
>>>>>> 
>>>>>> This assumes that forwarded packets don't count toward the idle timeout, which they shouldn't.  We might need to say that in the draft.
>>>>>> 
>>>>>> --Ben
>>>>>> From: Masque <[e-mail] <mailto:[e-mail]>> on behalf of Martin Duke <[e-mail] <mailto:[e-mail]>>
>>>>>> Sent: Friday, October 20, 2023 1:47 PM
>>>>>> To: Ben Schwartz <[e-mail] <mailto:[e-mail]>>
>>>>>> Cc: MASQUE <[e-mail] <mailto:[e-mail]>>
>>>>>> Subject: Re: [Masque] Forwarded Mode infinite loops
>>>>>>  
>>>>>> Ben,
>>>>>> 
>>>>>> Replies inline.
>>>>>> 
>>>>>> On Fri, Oct 20, 2023 at 9:53 AM Ben Schwartz <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>> Loops!  This is a fun attack.  It seems like there are probably lots of loopy configurations if you have access to two or more proxies.
>>>>>> 
>>>>>> Most of the encryption discussion for forwarded mode has focused on unauthenticated encryption, which wouldn't help here.
>>>>>> 
>>>>>> I have not seen a written proposal in this space, but if the encryption scheme has no authentication properties it doesn't mitigate pretty basic DoS attacks. Perhaps this attack creates a new design requirement for encryption?
>>>>>>  
>>>>>> Changing who chooses the Connection IDs could help, but proxy-chosen CIDs are not necessarily high entropy, so there's no clear guarantee.
>>>>>> 
>>>>>> IIUC your contention is that this arises when P3 and P2 both happen to select CID1 as the virtual connection ID. A well-intentioned client would therefore register CID1 to P1 twice, and P1 would reject it as a duplicate in one of those instances, avoiding a loop.
>>>>>> 
>>>>>> However, a malicious client could note the conflict and just register CID3 to P1 instead, creating the same attack.
>>>>>> 
>>>>>> I do think that if we try to fix this with proxy-chooses, there need to be some minimum size limits to avoid this kind of thing. If CIDs are 64 bits (or even 32!) I feel pretty good about this not being an exploitable attack.
>>>>>>  
>>>>>> Ultimately, I'm not sure this matters.  A hostile client can impose pretty much arbitrarily amplified load on the forwarder anyway, e.g. by downloading a large file and forging ACKs to dropped packets so that congestion control ramps up to infinity.  Forwarders will need defenses to rate-limit or cap abusive clients.  That said, it's probably worth fixing if there's an easy solution.
>>>>>> 
>>>>>> I agree there are other attacks -- in fact, I have other amplification attacks in mind, which will write about another day.
>>>>>> 
>>>>>> What is particularly pernicious about this attack is that a single attacker packet results in infinity packets in the proxy path. This flow can be "rate limited" but the client can go away and this persists forever.
>>>>>> 
>>>>>> Buit 
>>>>>>  
>>>>>> 
>>>>>> --Ben
>>>>>> From: Masque <[e-mail] <mailto:[e-mail]>> on behalf of Martin Duke <[e-mail] <mailto:[e-mail]>>
>>>>>> Sent: Wednesday, October 18, 2023 5:25 PM
>>>>>> To: MASQUE <[e-mail] <mailto:[e-mail]>>
>>>>>> Subject: [Masque] Forwarded Mode infinite loops
>>>>>>  
>>>>>> Hello MASQUE,
>>>>>> 
>>>>>> (with no hats on)
>>>>>> 
>>>>>> I was playing with some annoying but manageable amplification attacks on CONNECT-UDP forwarding mode. Those will have to wait for another day, because the current draft has a fatal vulnerability, though one that we can fix pretty easily.
>>>>>> 
>>>>>> If I've gotten something incorrect, please say so! It wouldn't be the first time.
>>>>>> 
>>>>>> Recall that for forwarded mode, clients notify the proxy of the connection ID they are advertising to the target, and in the same capsule propose a "virtual connection ID" for use on the proxy-client path. (IIUC, this virtual CID is to prevent association of the packets on either end of the proxy).
>>>>>> 
>>>>>> I reiterate that we are talking about the CLIENT connection ID and its virtual counterparts, not the target.
>>>>>> 
>>>>>> Consider the following very stupid topology that a client C could construct to target T via various proxies. Crucially, P2 and P3 are from the same provider and share a VIP.* This wrinkle eliminates some really simple mitigations for what I'm about to describe.
>>>>>> 
>>>>>> C -> P1 -> P2 -> P1 -> P3 ->  T
>>>>>> 
>>>>>> An otherwise well-intentioned client would advertise client Connection ID CID0 to T, and via REGISTER_CLIENT_CID capsules might set up the rest like so:
>>>>>> 
>>>>>> Nodes        C     P1     P2    P1    P3     T
>>>>>> CID on wire    CID4   CID3  CID2  CID1   CID0
>>>>>> 
>>>>>> So far, so good. A packet that leaves T will be destined for CID0. The destination CID will be rewritten in turn, arriving at C with CID4.
>>>>>> 
>>>>>> However, a malicious client can register CIDs with the proxies as follows:
>>>>>> Nodes        C     P1            P2      P1      P3     T
>>>>>> CID on wire    CID4   CID3 / CID1    CID2    CID1   CID0
>>>>>> 
>>>>>> The only change is that P2 is "incorrectly" configured by the client to rewrite incoming CID2 as CID1. As far as P2 is concerned, everything is seems legit unless it is sharing enormous amounts of CID state with its partner P3. Everything else is unchanged.
>>>>>> 
>>>>>> Thus, a packet leaving T with CID0 will be rewritten to CID1 and routed to P1;  P1 will then route it to P2 with CID2. Then that gets routed back to P1 with CID1, and the P1-P2-P1 loop continues to infinity. The attacker can put as many proxy hops as it likes in that chain.
>>>>>> 
>>>>>> There are a few pretty simple things that can break this doom loop.
>>>>>> - Do encryption at each MASQUE hop, as previously proposed. This verifies the previous hop of the packet and drops those where the CID does not have the correct origin.
>>>>>> - Have the proxy provide the virtual CID instead of the client. Client control is the root of these problems.
>>>>>> - Eliminate virtual client CIDs and just take the L on linkability in that case
>>>>>> 
>>>>>> Martin
>>>>>> 
>>>>>> * Interestingly, this attack requires at least one proxy in the path (P1 in this case) to not be load balanced or have an easily manipulated load balancer, so that two QUIC connection attempts can with high probability go to the same device.
>>> -- 
>>> Masque mailing list
>>> [e-mail] <mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/masque
> -- 
> Masque mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/masque