Re: Blocking packets from suspicious ports

[Christian Huitema <[e-mail]>]

I suppose we could start writing the advice in a draft. One more point 
below.

On 5/3/2022 9:09 PM, Willy Tarreau wrote:
> Hi Christian,
>
> On Tue, May 03, 2022 at 05:42:56PM -0700, Christian Huitema wrote:
>> As I said, there appear to be two concerns. One is to protect the local
>> server from floods, and, as you say, this is much better done by blocking
>> the suspicious traffic ahead of the server. The other is, provide at least a
>> modicum of defense against attacks that use the QUIC server as a reflector.
>> For example:
>>
>> * attacker sends a "carefully crafted" UDP packet to QUIC server, spoofing
>> the source address and port of a target,
>>
>> * QUIC server processes the packet and produces the expected "response"
>>
>> * target attempts to process the packet and something bad happens.
>>
>> The concern here is not so much volumetric attacks, which can be mitigated
>> with a combination of not amplifying traffic and carefully using retry. We
>> are also concerned about "Request Forgery" attacks, in which the "response"
>> activates a bug in the target. For example, QUIC sends a long packet to a
>> poorly written game server, and the game server crashes when trying to
>> process that packet.
> Sure but this problem has been existing for as long as UDP services were
> reachable on the net, using ECHO, CHARGEN or RPCSS as generators or
> reflectors, and a variety of other services as targets.
Yes indeed. However, see below on the extra attack avenue bouncing 
through clients when pretending to "move from anycast to unicast".
>> There may also be variants of that attack in which an
>> attacking server bounces packets through a QUIC client.
> This is much less likely to happen since most clients nowadays are behind
> stateful NAT, so the attacking server must use the same source address as
> an existing connection whose server will be attacked. However, I predict
> that we'll see new privacy attacks based on IP ID generation so that an
> attacker can try to guess whether or not a client currently has an
> established QUIC connection with a given server. The principle will be
> the same as with TCP and will consist in trying to forge a packet as
> coming from the supposed server and seeing if the client responds/retries,
> causing a change in the IP ID generation of subsequent packets. I think
> by now most operating systems are safe regarding this, but we'll see in
> the long term.

QUIC includes a feature to facilitate moving a connection from an 
anycast address to a unicast address. The client finds the anycast 
address through the DNS, contacts the server, learns a "preferred server 
address" during the handshake, and tries validating that preferred 
address before using it for the rest of the connection duration, without 
risk losing the connection to a routing event that redirects anycast to 
a different server. Which is nice, but of course a villainous server 
could feed a bogus preferred address. That's the client bounce for you. 
All the server packets before the bounce are sent with the expected 5 
tuple. And of course the "preferred address" is sent using encrypted 
packets.


>> Clearly, servers that can be crashed by a few magic packets have no business
>> being reachable from the Internet. But the concern here is that by spoofing
>> the source address, an attacker located outside of the server network can
>> reflect and attack towards a vulnerable server inside that network, even if
>> firewalls isolate that vulnerable server from the Internet.
> This one is already solved by anti-spoofing at the edge and is not
> specific to QUIC, and it's valid for other protocols (including ICMP
> and TCP). If you do not filter your own sources at the edge, your
> internal network is already exposed to attacks and scans. Such attacks
> have been widely known and documented for 2 decades now, and similarly
> to what you mentioned above, infrastructure that doesn't filter its own
> source at the edge has no business being reachable from the net.
>
> But it's very possible that we'll see a reload of such attacks with
> QUIC connection attempts from ports 514 (syslog), 162 (snmp trap) or
> even 161 (snmp) as an attempt to catch such misconfigured networks.
> Again this is another reason for servers not to accept connections
> from unprivileged ports.
>
>>> I would suggest that port filtering at the application layer is only
>>> used to decide whether to respond or not (i.e. should I send a retry
>>> for a packet that parses correctly). For example it could be suggested
>>> that packets coming from suspicious but valid source ports ought to
>>> be double-checked with a retry to preserve local resources. But that
>>> will not be used to save CPU anyway.
>> Yes. Makes sense. But there are four ways by which QUIC can bounce packets:
>>
>> * Initial packets from the client will cause the server to send a flight of
>> up to 3 Handshake packets. The Retry mechanism could be used there.
>>
>> * Initial packets from the client with an unsupported version number cause
>> the server to send a Version Negotiation packet. These are short packets,
>> but up to 256 bytes of the VN packet content can be predetermined by the
>> attacker.
> 256 bytes are sufficient to cause a large DNS or NTP request, so what could
> happen is an attacker sending a packet from port 53 or 123 of a server
> known for responding large packets, to a victim QUIC server, that will
> respond to this server and cause a large response to be sent to it. As
> such the attacker will only need to send 256 bytes to the victim, and
> the victim will itself request the extra few kB from another location.
> This is another form of amplification attack, which is properly addressed
> by filtering these invalid source ports.
>
>> * 1RTT packets sent to a server with a unknown Connection ID may cause the
>> server to send a Stateless Reset packet. The Stateless Reset packet must be
>> shorter than the incoming packet, so there is no amplification. The content
>> is hard to predict, but who knows.
>>
>> * After a connection has been established and verified, either party can
>> send a "path validation" packet from a "new" address and port. Clients or
>> servers will respond with a path validation packet of their own, trying to
>> reach the new address. The validation attempt could be repeated multiple
>> time, typically 3 times. At least 20 bytes at the beginning of the packet
>> can be controlled by the attacker, and possibly many more.
>>
>> If an application just drops packets from a suspicious port, it mitigates
>> all 4 avenues of attack. If it want precision instead, it has to write
>> specific code at four locations. RFC 9000 details these attacks, but mostly
>> says "protect using ingress filtering, per BCP38".
> There *will* inevitably be some problems at the beginning, but one good
> thing is that solutions are reasonably simple, can instantly be deployed
> in field using port filtering, and will quickly land in the code itself
> because the various stack authors have full control of their code.

Port filtering at the edge will not catch the "preferred address attack" 
in which the server proposes a non-anycast address to the client, so I 
guess QUIC stacks will have to protect that. As in, if a server says 
"please move the connection to 10.0.0.1:53", the client should just 
decline and stick to the anycast address. And yes, using a configurable 
list makes sense.


>
> It could have been great to decide that the first byte in the protocol
> should be random. This would have significantly voided most possibilities
> of abuse by making the first byte non-predictable nor controllable. But
> this is easy to say afterwards and it could come with its own set of
> problems (e.g. fingerprinting from the random sequence) ;-)

That one will have to wait for V2. Or V3, actually.

>>> Maintaining a public list of well-known suspicious ports can be both
>>> helpful and dangerous. It's helpful in that it shows implementers that
>>> some ranges should never appear as they're normally reserved for
>>> listening services, that some numbers are well known and widely
>>> deployed, and that some are less common but appear anywhere, indicating
>>> that a range is not sufficient. This should help design a flexible
>>> filtering mechanism. But it can also be dangerous if applied as-is:
>>> this list cannot remain static as new entries may have to be added
>>> within a few minutes to hours hours and each addition will cause extra
>>> breakage, thus some previous ones will likely be dropped after a
>>> previous service stopped being widely attacked. I.e. the list in
>>> question likely needs to be accessible by configuration in field and
>>> not be hard-coded.
>> AFAIK, such lists are hard coded in quite a few implementations. Even if
>> they are provided in a configuration file, changing the file requires
>> restarting the server. So "within a few minutes" may be problematic.
> It's not a problem anymore once under heavy attack, believe me ;-)
I believe you. Memories of Blaster and all that.
> Usually production management rules are very strict, but once a site is
> under DDoS and zero useful traffic passes, all the rules suddenly change
> and you don't even need a permission anymore to change a config file and
> restart a process multiple times until the site recovers! The worst
> moment is when it starts to work again and you would like to clean up
> all the mess in your config files but you're not allowed to touch
> anything anymore, as if your actions were the ones that caused the
> problem in the first place.
>
>>> Other approaches could work fairly well, such as keeping a rate counter
>>> of incoming packets per source port (e.g. unparsable or initial packets)
>>> which, above a moderate value, would serve to send retry, and above a
>>> critical value, can be used to decide to block the port (possibly earlier
>>> in the chain). This remains reasonably cheap to implement, though it may
>>> end up causing some flapping as the rate will fall after the port is
>>> blocked upstream, which may lead to it being reopened before the attack
>>> is finished.
>> Yes. Should that really be "per source port" or "per source IP + port"?
> Per port should be sufficient. There's no reason from the server side that
> multiple clients collectively decide to use the same source port to send
> legit traffic. And using the port alone will allow to quicky detect a new
> internet-wide attack coming from a randomly vulnerable service (IoT
> gateways, ToIP etc) regardless of their address. I don't know how this
> would be distributed, but I guess that if a source port is responsible
> for 100 times more traffic than all other one combined, it will usually
> be a good indicator that you don't want to deal with that traffic anymore.
I would expect quite a bit of legit server-to-server traffic on ports 
443 or 853, but yes, keeping a dynamic list of most abused ports makes 
sense. Although that's mostly an issue for volumetric attacks, and host 
based defenses are not too efficient there, but still it feels better 
than just receiving the attack and waiting until it stops.
>>> I think we'll discover new fun things over time and will learn new
>>> attacks and workarounds as deployments grow.
>> I have not heard of actual attacks "in the wild", but who knows...
> It's too early, stacks are still evolving and the level of deployment
> is not high yet. That doesn't mean nobody is trying to elaborate new
> attacks. For example yesterday on my home servers which do not advertise
> QUIC I received one short packet from port 443 to port 443 from a host
> named "scanners.labs.something", and a 1238-byte one from a random port
> to 443 from a host named "scan-19a.somethingelse". Thus it shows that
> studies are already in progress. We just need to be patient :-)


We probably will not have to wait too long...

Thanks for the feedback.

-- Christian Huitema