Re: [Masque] Forwarded Mode infinite loops

[Martin Duke <[e-mail]>]

That is pretty much a photo of my whiteboard when crafting the
original email!

ISTM the design has to give up on one of three principles and/or
operational degrees of freedom:
(1) Receiver picks its connection ID.
(2) Proxies don't need to use server preferred address to use this feature
(3) Proxies don't have to open a new target-facing UDP socket for each
CONNECT (unlike CONNECT-UDP!)

IIUC you are proposing eliminating (2), which to me sounds like the biggest
constraint to add. But you guys are the authors, so do what you think is
best.

I also have a sense that (2) is a point solution to the only attack someone
has managed to assemble, and that other problems of this nature are out
there. But that's just conjecture at this point.

Martin

On Tue, Oct 24, 2023 at 11:52 AM David Schinazi <[e-mail]>
wrote:

> re: Martin and Tommy, thanks for the clarification about ingress vs egress
> - that's super helpful. My whiteboard was starting to look like this:
>
> https://i.kym-cdn.com/entries/icons/original/000/022/524/pepe_silvia_meme_banner.jpg
>

At a more fundamental level, the issue stems from the fact that we're
> breaking one of the rules of QUIC. In QUIC, the general mindset is that the
> receiver of a Destination Connection ID is the one that picks this CID. The
> draft violates that in a few ways:
> * The Client Connection ID is picked by the client even though it's
> received by the proxy egress. This one's OK because the proxy gets to
> accept or reject the proposed CID.
> * The Target Connection ID is picked by the target, but communicated to
> the proxy by the client, so the client could lie. I haven't been able to
> mount an attack using that fact though.
> * The Virtual Client CID is picked by the client. That's perfectly fine in
> a single instance of this draft, but once you chain two of these
> (C-P1-P2-T) then the client gets to pick the Virtual Client CID that P2
> sends to P1. This is the one that causes the attack here, Client is picking
> a CID that P1 receives.
>
> I'm not sure that having the proxy pick the Virtual Client CID is a good
> fix here because that also violates the QUIC rule of receiver picks CIDs
> and could lead to other issues where the proxy is the attacker.
>


>
> A mitigation we've discussed is the fact that the proxy could look at the
> target ingress IP when routing. That breaks when multiple proxies share an
> ingress IP. Perhaps we can prevent that by requiring something similar to
> server preferred address where the proxy can only use forwarded mode on an
> ingress IP that is unique to them?
>


>
> David
>
> On Tue, Oct 24, 2023 at 9:57 AM Martin Duke <[e-mail]>
> wrote:
>
>> That's a good question!
>>
>> IIUC, CONNECT-IP does decrement the TTL, but CONNECT-UDP does not because
>> it is not preserving the IP header (The CONNECT-IP draft mentions TTL and
>> CONNECT-UDP does not). The quic-proxy draft doesn't say anything about
>> this, so I imagine that rule still applies.
>>
>> On Tue, Oct 24, 2023 at 6:43 AM Ian Swett <[e-mail]> wrote:
>>
>>> I assume each hop would decrement the TTL?  If so, wouldn't this be a
>>> loop, but not infinite?
>>>
>>> Ian
>>>
>>> On Mon, Oct 23, 2023, 11:01 PM Tommy Pauly <tpauly=
>>> [e-mail]> wrote:
>>>
>>>> It was useful for me to draw this out in (unnecessary) detail to show
>>>> how this only needs the ingress IP to be shared, not the egress IP.
>>>>
>>>> To help go through this, let me fill in the example you gave with more
>>>> details about IPs and ports — since the forwarding mappings depend on these
>>>> “sockets”.
>>>>
>>>> Let’s start with a case where the addresses aren’t being shared.
>>>>
>>>> *C *has address 1111::1
>>>> *P1* has ingress address 2222::2, and egress address 3333::3
>>>> *P2* has ingress address 4444::4, and egress address 5555::5
>>>> *P3* has ingress address 6666::6, and egress address 7777::7
>>>> *T* has ingress address 8888::8
>>>>
>>>> *C *is set up to receive CID4 on 1111::1 port 10000 from 2222::2 port
>>>> 443
>>>> *P1 *is set up to receive CID3 on 3333::3 port 20000 from 4444::4 port
>>>> 443, and forward it as CID4 to 1111::1 port 10000 from 2222::2 port 443
>>>> *P2 *is set up to receive CID2 on 5555::5 port 30000 from 2222::2 port
>>>> 443, and forward it as CID3 to 3333::3 port 20000 from 4444::4 port 443
>>>> *P1 *is set up to receive CID1 on 3333::3 port 40000 from 6666::6 port
>>>> 443, and forward it as CID2 to 5555::5 port 30000 from 2222::2 port 443
>>>> *P3 *is set up to receive CID0 on 7777::7 port 50000 from 8888::8 port
>>>> 443, and forward it as CID1 to 3333::3 port 40000 from 6666::6 port 443
>>>> *T *sends using CID0 to 7777::7 port 50000 from 8888::8 port 443
>>>>
>>>> In this case, since there is no address sharing, if the client tells
>>>> *P2* to instead map from (CID2 on 5555::5 port 30000 from 2222::2 port
>>>> 443) to (CID1 to 3333::3 port 20000 from 4444::4 port 443), the
>>>> connection will break, but there will be no loop, since *P1* won’t
>>>> allow receiving CID1 on that socket.
>>>>
>>>>
>>>> Then, let’s explore the case of shared addresses. If we only have P2
>>>> and P3 share an ingress VIP, and P1 reuses sockets on egress, we have the
>>>> following:
>>>>
>>>> *C *has address 1111::1
>>>> *P1* has ingress address 2222::2, and egress address 3333::3
>>>> *P2* has ingress address 4444::4, and egress address 5555::5
>>>> *P3* has ingress address 4444::4, and egress address 7777::7
>>>> *T* has ingress address 8888::8
>>>>
>>>> *C *is set up to receive CID4 on 1111::1 port 10000 from 2222::2 port
>>>> 443
>>>> *P1 *is set up to receive CID3 on 3333::3 port 20000 from 4444::4 port
>>>> 443, and forward it as CID4 to 1111::1 port 10000 from 2222::2 port 443
>>>> *P2 *is set up to receive CID2 on 5555::5 port 30000 from 2222::2 port
>>>> 443, and forward it as CID3 to 3333::3 port 20000 from 4444::4 port 443
>>>> *P1 *is set up to receive CID1 on 3333::3 port 20000 from 4444::4 port
>>>> 443, and forward it as CID2 to 5555::5 port 30000 from 2222::2 port 443
>>>> *P3 *is set up to receive CID0 on 7777::7 port 50000 from 8888::8 port
>>>> 443, and forward it as CID1 to 3333::3 port 20000 from 4444::4 port 443
>>>> *T *sends using CID0 to 7777::7 port 50000 from 8888::8 port 443
>>>>
>>>> Then, if the client tells *P2* to instead map from (CID2 on 5555::5
>>>> port 30000 from 2222::2 port 443) to (CID1 to 3333::3 port 20000 from
>>>> 4444::4 port 443), the loop occurs because there are two mappings for
>>>> receiving on *P1 *that differ only in CID.
>>>>
>>>> Tommy
>>>>
>>>> On Oct 23, 2023, at 7:08 PM, Martin Duke <[e-mail]>
>>>> wrote:
>>>>
>>>> The reason I put in the point about VIPs is that P1 cannot distinguish
>>>> between a packet from P2 and P3 if they share an ingress VIP
>>>>
>>>> If they're from different IP addresses it's sufficient for P1 to check
>>>> the source IP of packets arriving from the target
>>>>
>>>> Does that answer your question?
>>>>
>>>>
>>>> On Mon, Oct 23, 2023, 18:34 David Schinazi <[e-mail]>
>>>> wrote:
>>>>
>>>>> Hi Martin,
>>>>>
>>>>> I don't understand how the fact that P2 and P3 share a VIP impacts the
>>>>> attack. Thinking about this some more, maybe the issue is that we mean
>>>>> different things when we say VIP? In my mind:
>>>>>
>>>>> * a "VIP" is an (IP address, UDP port) tuple. The V stands for virtual
>>>>> in the
>>>>>   sense that two separate servers can both have the same IP due to
>>>>> anycast.
>>>>>
>>>>> * an "ingress VIP" is the VIP that the client sends packets to. In the
>>>>> document
>>>>>   this is called a "client-facing socket".
>>>>>
>>>>> * an "egress VIP" is the VIP that the target sends packets to. In the
>>>>> document
>>>>>   this is called a "target-facing socket".
>>>>>
>>>>> Ingress VIPs are very common in practice. Many CDNs will use anycast
>>>>> and
>>>>> have an ingress VIP that represents a fleet of frontends. Egress VIPs
>>>>> however,
>>>>> don't make any sense to me, because you'd never want anycast on egress
>>>>> IPs.
>>>>>
>>>>> Could you help reframe the P2/P3 shared VIP point in terms of ingress
>>>>> and
>>>>> egress VIPs please?
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>>
>>>>> On Fri, Oct 20, 2023 at 3:00 PM Martin Duke <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> As I continue to think about this, I'm not sure that
>>>>>> authentication even solves the problem. The client could authenticate with
>>>>>> each client, sure, but to P1, P2 and P3 are essentially targets. Targets,
>>>>>> obviously will not do anything special just because this is a connection
>>>>>> over MASQUE, so there cannot be a special mechanism for P1 to verify
>>>>>> authenticity of packets from the target, unless it has the target-client
>>>>>> session keys.
>>>>>>
>>>>>> On Fri, Oct 20, 2023 at 12:14 PM Martin Duke <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>> I should add that another way to thwart this attack is to require
>>>>>>> the proxy to open a new target-facing socket for every client request. Then
>>>>>>> the packet incoming to P1 would be disambiguated by the destination
>>>>>>> IP/port. Not opening a new socket is currently allowed in the spec, which
>>>>>>> IIUC is a change from CONNECT-UDP.
>>>>>>>
>>>>>>> So the proxy vulnerability occurs if
>>>>>>> (1) P2/P3 share a VIP,
>>>>>>> (2) P1 is not load balanced, so that the attacker can directly
>>>>>>> target it, and
>>>>>>> (3) P1 is not opening unique target-facing sockets for each request,
>>>>>>> even though the target IP/port are the same.
>>>>>>>
>>>>>>> On Fri, Oct 20, 2023 at 11:56 AM Martin Duke <
>>>>>>> [e-mail]> wrote:
>>>>>>>
>>>>>>>> Forwarded packets from the client SHOULD count. The draft is silent
>>>>>>>> about forwarded packets from the target.
>>>>>>>>
>>>>>>>> But fair enough, the client needs to do just enough to keep it
>>>>>>>> alive.
>>>>>>>>
>>>>>>>> On Fri, Oct 20, 2023 at 11:15 AM Ben Schwartz <[e-mail]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm not sure the client can actually "go away".  If the HTTP
>>>>>>>>> connection ends, the forwarding sessions will also end, so this can be
>>>>>>>>> addressed by setting a QUIC max_idle_timeout.
>>>>>>>>>
>>>>>>>>> This assumes that forwarded packets don't count toward the idle
>>>>>>>>> timeout, which they shouldn't.  We might need to say that in the draft.
>>>>>>>>>
>>>>>>>>> --Ben
>>>>>>>>> ------------------------------
>>>>>>>>> *From:* Masque <[e-mail]> on behalf of Martin Duke
>>>>>>>>> <[e-mail]>
>>>>>>>>> *Sent:* Friday, October 20, 2023 1:47 PM
>>>>>>>>> *To:* Ben Schwartz <[e-mail]>
>>>>>>>>> *Cc:* MASQUE <[e-mail]>
>>>>>>>>> *Subject:* Re: [Masque] Forwarded Mode infinite loops
>>>>>>>>>
>>>>>>>>> Ben, Replies inline. On Fri, Oct 20, 2023 at 9: 53 AM Ben Schwartz
>>>>>>>>> <bemasc@ meta. com> wrote: Loops! This is a fun attack. It seems
>>>>>>>>> like there are probably lots of loopy configurations if you have access to
>>>>>>>>> two or more proxies. Most of the
>>>>>>>>> Ben,
>>>>>>>>>
>>>>>>>>> Replies inline.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 20, 2023 at 9:53 AM Ben Schwartz <[e-mail]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Loops!  This is a fun attack.  It seems like there are probably
>>>>>>>>> lots of loopy configurations if you have access to two or more proxies.
>>>>>>>>>
>>>>>>>>> Most of the encryption discussion for forwarded mode has focused
>>>>>>>>> on unauthenticated encryption, which wouldn't help here.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have not seen a written proposal in this space, but if the
>>>>>>>>> encryption scheme has no authentication properties it doesn't mitigate
>>>>>>>>> pretty basic DoS attacks. Perhaps this attack creates a new design
>>>>>>>>> requirement for encryption?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Changing who chooses the Connection IDs could help, but
>>>>>>>>> proxy-chosen CIDs are not necessarily high entropy, so there's no clear
>>>>>>>>> guarantee.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> IIUC your contention is that this arises when P3 and P2 both
>>>>>>>>> happen to select CID1 as the virtual connection ID. A well-intentioned
>>>>>>>>> client would therefore register CID1 to P1 twice, and P1 would reject it as
>>>>>>>>> a duplicate in one of those instances, avoiding a loop.
>>>>>>>>>
>>>>>>>>> However, a malicious client could note the conflict and just
>>>>>>>>> register CID3 to P1 instead, creating the same attack.
>>>>>>>>>
>>>>>>>>> I do think that if we try to fix this with proxy-chooses, there
>>>>>>>>> need to be some minimum size limits to avoid this kind of thing. If CIDs
>>>>>>>>> are 64 bits (or even 32!) I feel pretty good about this not being an
>>>>>>>>> exploitable attack.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ultimately, I'm not sure this matters.  A hostile client can
>>>>>>>>> impose pretty much arbitrarily amplified load on the forwarder anyway, e.g.
>>>>>>>>> by downloading a large file and forging ACKs to dropped packets so that
>>>>>>>>> congestion control ramps up to infinity.  Forwarders will need defenses to
>>>>>>>>> rate-limit or cap abusive clients.  That said, it's probably worth fixing
>>>>>>>>> if there's an easy solution.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I agree there are other attacks -- in fact, I have other
>>>>>>>>> amplification attacks in mind, which will write about another day.
>>>>>>>>>
>>>>>>>>> What is particularly pernicious about this attack is that a single
>>>>>>>>> attacker packet results in infinity packets in the proxy path. This flow
>>>>>>>>> can be "rate limited" but the client can go away and this persists forever.
>>>>>>>>>
>>>>>>>>> Buit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --Ben
>>>>>>>>> ------------------------------
>>>>>>>>> *From:* Masque <[e-mail]> on behalf of Martin Duke
>>>>>>>>> <[e-mail]>
>>>>>>>>> *Sent:* Wednesday, October 18, 2023 5:25 PM
>>>>>>>>> *To:* MASQUE <[e-mail]>
>>>>>>>>> *Subject:* [Masque] Forwarded Mode infinite loops
>>>>>>>>>
>>>>>>>>> Hello MASQUE, (with no hats on) I was playing with some annoying
>>>>>>>>> but manageable amplification attacks on CONNECT-UDP forwarding mode. Those
>>>>>>>>> will have to wait for another day, because the current draft has a fatal
>>>>>>>>> vulnerability, though one that
>>>>>>>>> Hello MASQUE,
>>>>>>>>>
>>>>>>>>> (with no hats on)
>>>>>>>>>
>>>>>>>>> I was playing with some annoying but manageable amplification
>>>>>>>>> attacks on CONNECT-UDP forwarding mode. Those will have to wait for another
>>>>>>>>> day, because the current draft has a fatal vulnerability, though one that
>>>>>>>>> we can fix pretty easily.
>>>>>>>>>
>>>>>>>>> If I've gotten something incorrect, please say so! It wouldn't be
>>>>>>>>> the first time.
>>>>>>>>>
>>>>>>>>> Recall that for forwarded mode, clients notify the proxy of the
>>>>>>>>> connection ID they are advertising to the target, and in the same
>>>>>>>>> capsule propose a "virtual connection ID" for use on the proxy-client path.
>>>>>>>>> (IIUC, this virtual CID is to prevent association of the packets on either
>>>>>>>>> end of the proxy).
>>>>>>>>>
>>>>>>>>> I reiterate that we are talking about the CLIENT connection ID and
>>>>>>>>> its virtual counterparts, not the target.
>>>>>>>>>
>>>>>>>>> Consider the following very stupid topology that a client C could
>>>>>>>>> construct to target T via various proxies. Crucially, P2 and P3 are from
>>>>>>>>> the same provider and share a VIP.* This wrinkle eliminates some really
>>>>>>>>> simple mitigations for what I'm about to describe.
>>>>>>>>>
>>>>>>>>> C -> P1 -> P2 -> P1 -> P3 ->  T
>>>>>>>>>
>>>>>>>>> An otherwise well-intentioned client would advertise client
>>>>>>>>> Connection ID CID0 to T, and via REGISTER_CLIENT_CID capsules might set up
>>>>>>>>> the rest like so:
>>>>>>>>>
>>>>>>>>> Nodes        C     P1     P2    P1    P3     T
>>>>>>>>> CID on wire    CID4   CID3  CID2  CID1   CID0
>>>>>>>>>
>>>>>>>>> So far, so good. A packet that leaves T will be destined for CID0.
>>>>>>>>> The destination CID will be rewritten in turn, arriving at C with CID4.
>>>>>>>>>
>>>>>>>>> However, a malicious client can register CIDs with the proxies as
>>>>>>>>> follows:
>>>>>>>>> Nodes        C     P1            P2      P1      P3     T
>>>>>>>>> CID on wire    CID4   CID3 / CID1    CID2    CID1   CID0
>>>>>>>>>
>>>>>>>>> The only change is that P2 is "incorrectly" configured by the
>>>>>>>>> client to rewrite incoming CID2 as CID1. As far as P2 is concerned,
>>>>>>>>> everything is seems legit unless it is sharing enormous amounts of CID
>>>>>>>>> state with its partner P3. Everything else is unchanged.
>>>>>>>>>
>>>>>>>>> Thus, a packet leaving T with CID0 will be rewritten to CID1 and
>>>>>>>>> routed to P1;  P1 will then route it to P2 with CID2. Then that gets routed
>>>>>>>>> back to P1 with CID1, and the P1-P2-P1 loop continues to infinity. The
>>>>>>>>> attacker can put as many proxy hops as it likes in that chain.
>>>>>>>>>
>>>>>>>>> There are a few pretty simple things that can break this doom loop.
>>>>>>>>> - Do encryption at each MASQUE hop, as previously proposed. This
>>>>>>>>> verifies the previous hop of the packet and drops those where the CID does
>>>>>>>>> not have the correct origin.
>>>>>>>>> - Have the proxy provide the virtual CID instead of the client.
>>>>>>>>> Client control is the root of these problems.
>>>>>>>>> - Eliminate virtual client CIDs and just take the L on linkability
>>>>>>>>> in that case
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>> * Interestingly, this attack requires at least one proxy in the
>>>>>>>>> path (P1 in this case) to not be load balanced or have an easily
>>>>>>>>> manipulated load balancer, so that two QUIC connection attempts can with
>>>>>>>>> high probability go to the same device.
>>>>>>>>>
>>>>>>>>> --
>>>>>> Masque mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/masque
>>>>>>
>>>>> --
>>>> Masque mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/masque
>>>>
>>>>
>>>> --
>>>> Masque mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/masque
>>>>
>>>