Re: Blocking packets from suspicious ports

[Christian Huitema <[e-mail]>]

Thanks, Willy. That's exactly the kind of analysis I was trying to elicit.

Comments in line.

On 5/2/2022 8:23 PM, Willy Tarreau wrote:
> On Mon, May 02, 2022 at 02:27:15PM -0700, Paul Vixie wrote:
>> a partially managed (not fully transparent) network (public or private) can
>> be expected to implement port-based inbound UDP blocking of the kind you
>> describe. the set of ports will be dynamic, updated during attacks. not
>> something to be hardcoded or "set and forget".
> I think that's the most important aspect, and for having participated many
> times to fightng DDoS I agree with this. There are obvious common sanity
> rules that are often applied based on what is possible/available:
>    - have a (set of) local DNS and NTP server(s) that are the only ones
>      susceptible of receiving UDP from privilege ports (< 1024) and have
>      all other servers use DNS and NTP from there
>    - have the edge components (routers, L3 switches) block all inbound
>      UDP from privileged and well-known source ports to regular servers;
>      this list may evolve during attacks
>    - have all servers locally refine their blocking list at the kernel
>      level (netfilter, BPF, PF and so on etc) preventing cross-protocol
>      communication (53->123, 123->53 etc).
>    - when possible, refine filtering by source/destination ports based
>      on the expected protocol (e.g. verify that traffic to port 53 looks
>      like a DNS request, from port 53 looks like a DNS response, from
>      123 looks like an NTP response, with special cases for 53->53 or
>      123->123)
>
> If the wrong packet is delivered to the userspace application, you lose
> anyway as most of the harm caused by network stack traversal by a packet
> was already done. Worse once a packet passes through, it's often trivial
> for the attacker to repeat it and flood the application. A good rule of
> thumb is to count on 1 million packets per second delivered to the
> application, per CPU core. Of course it will vary a lot between systems
> and the filtering in place, but the order of magnitude is there. This
> means that a 100G NIC can keep 148 cores busy just delivering bad
> packets to be dropped by the application.

As I said, there appear to be two concerns. One is to protect the local 
server from floods, and, as you say, this is much better done by 
blocking the suspicious traffic ahead of the server. The other is, 
provide at least a modicum of defense against attacks that use the QUIC 
server as a reflector. For example:

* attacker sends a "carefully crafted" UDP packet to QUIC server, 
spoofing the source address and port of a target,

* QUIC server processes the packet and produces the expected "response"

* target attempts to process the packet and something bad happens.

The concern here is not so much volumetric attacks, which can be 
mitigated with a combination of not amplifying traffic and carefully 
using retry. We are also concerned about "Request Forgery" attacks, in 
which the "response" activates a bug in the target. For example, QUIC 
sends a long packet to a poorly written game server, and the game server 
crashes when trying to process that packet. There may also be variants 
of that attack in which an attacking server bounces packets through a 
QUIC client.

Clearly, servers that can be crashed by a few magic packets have no 
business being reachable from the Internet. But the concern here is that 
by spoofing the source address, an attacker located outside of the 
server network can reflect and attack towards a vulnerable server inside 
that network, even if firewalls isolate that vulnerable server from the 
Internet.

> I would suggest that port filtering at the application layer is only
> used to decide whether to respond or not (i.e. should I send a retry
> for a packet that parses correctly). For example it could be suggested
> that packets coming from suspicious but valid source ports ought to
> be double-checked with a retry to preserve local resources. But that
> will not be used to save CPU anyway.

Yes. Makes sense. But there are four ways by which QUIC can bounce packets:

* Initial packets from the client will cause the server to send a flight 
of up to 3 Handshake packets. The Retry mechanism could be used there.

* Initial packets from the client with an unsupported version number 
cause the server to send a Version Negotiation packet. These are short 
packets, but up to 256 bytes of the VN packet content can be 
predetermined by the attacker.

* 1RTT packets sent to a server with a unknown Connection ID may cause 
the server to send a Stateless Reset packet. The Stateless Reset packet 
must be shorter than the incoming packet, so there is no amplification. 
The content is hard to predict, but who knows.

* After a connection has been established and verified, either party can 
send a "path validation" packet from a "new" address and port. Clients 
or servers will respond with a path validation packet of their own, 
trying to reach the new address. The validation attempt could be 
repeated multiple time, typically 3 times. At least 20 bytes at the 
beginning of the packet can be controlled by the attacker, and possibly 
many more.

If an application just drops packets from a suspicious port, it 
mitigates all 4 avenues of attack. If it want precision instead, it has 
to write specific code at four locations. RFC 9000 details these 
attacks, but mostly says "protect using ingress filtering, per BCP38".

>
> Maintaining a public list of well-known suspicious ports can be both
> helpful and dangerous. It's helpful in that it shows implementers that
> some ranges should never appear as they're normally reserved for
> listening services, that some numbers are well known and widely
> deployed, and that some are less common but appear anywhere, indicating
> that a range is not sufficient. This should help design a flexible
> filtering mechanism. But it can also be dangerous if applied as-is:
> this list cannot remain static as new entries may have to be added
> within a few minutes to hours hours and each addition will cause extra
> breakage, thus some previous ones will likely be dropped after a
> previous service stopped being widely attacked. I.e. the list in
> question likely needs to be accessible by configuration in field and
> not be hard-coded.
AFAIK, such lists are hard coded in quite a few implementations. Even if 
they are provided in a configuration file, changing the file requires 
restarting the server. So "within a few minutes" may be problematic.
> Other approaches could work fairly well, such as keeping a rate counter
> of incoming packets per source port (e.g. unparsable or initial packets)
> which, above a moderate value, would serve to send retry, and above a
> critical value, can be used to decide to block the port (possibly earlier
> in the chain). This remains reasonably cheap to implement, though it may
> end up causing some flapping as the rate will fall after the port is
> blocked upstream, which may lead to it being reopened before the attack
> is finished.
Yes. Should that really be "per source port" or "per source IP + port"?
>
> I think we'll discover new fun things over time and will learn new
> attacks and workarounds as deployments grow.

I have not heard of actual attacks "in the wild", but who knows...

-- Christian Huitema