Re: Blocking packets from suspicious ports

[Willy Tarreau <[e-mail]>]

On Mon, May 02, 2022 at 02:27:15PM -0700, Paul Vixie wrote:
> a partially managed (not fully transparent) network (public or private) can
> be expected to implement port-based inbound UDP blocking of the kind you
> describe. the set of ports will be dynamic, updated during attacks. not
> something to be hardcoded or "set and forget".

I think that's the most important aspect, and for having participated many
times to fightng DDoS I agree with this. There are obvious common sanity
rules that are often applied based on what is possible/available:
  - have a (set of) local DNS and NTP server(s) that are the only ones
    susceptible of receiving UDP from privilege ports (< 1024) and have
    all other servers use DNS and NTP from there
  - have the edge components (routers, L3 switches) block all inbound
    UDP from privileged and well-known source ports to regular servers;
    this list may evolve during attacks
  - have all servers locally refine their blocking list at the kernel
    level (netfilter, BPF, PF and so on etc) preventing cross-protocol
    communication (53->123, 123->53 etc).
  - when possible, refine filtering by source/destination ports based
    on the expected protocol (e.g. verify that traffic to port 53 looks
    like a DNS request, from port 53 looks like a DNS response, from
    123 looks like an NTP response, with special cases for 53->53 or
    123->123)

If the wrong packet is delivered to the userspace application, you lose
anyway as most of the harm caused by network stack traversal by a packet
was already done. Worse once a packet passes through, it's often trivial
for the attacker to repeat it and flood the application. A good rule of
thumb is to count on 1 million packets per second delivered to the
application, per CPU core. Of course it will vary a lot between systems
and the filtering in place, but the order of magnitude is there. This
means that a 100G NIC can keep 148 cores busy just delivering bad
packets to be dropped by the application.

I would suggest that port filtering at the application layer is only
used to decide whether to respond or not (i.e. should I send a retry
for a packet that parses correctly). For example it could be suggested
that packets coming from suspicious but valid source ports ought to
be double-checked with a retry to preserve local resources. But that
will not be used to save CPU anyway.

Maintaining a public list of well-known suspicious ports can be both
helpful and dangerous. It's helpful in that it shows implementers that
some ranges should never appear as they're normally reserved for
listening services, that some numbers are well known and widely
deployed, and that some are less common but appear anywhere, indicating
that a range is not sufficient. This should help design a flexible
filtering mechanism. But it can also be dangerous if applied as-is:
this list cannot remain static as new entries may have to be added
within a few minutes to hours hours and each addition will cause extra
breakage, thus some previous ones will likely be dropped after a
previous service stopped being widely attacked. I.e. the list in
question likely needs to be accessible by configuration in field and
not be hard-coded.

Other approaches could work fairly well, such as keeping a rate counter
of incoming packets per source port (e.g. unparsable or initial packets)
which, above a moderate value, would serve to send retry, and above a
critical value, can be used to decide to block the port (possibly earlier
in the chain). This remains reasonably cheap to implement, though it may
end up causing some flapping as the rate will fall after the port is
blocked upstream, which may lead to it being reopened before the attack
is finished.

I think we'll discover new fun things over time and will learn new
attacks and workarounds as deployments grow.

Willy