Re: Blocking packets from suspicious ports

[Willy Tarreau <[e-mail]>]

Hi Christian,

On Tue, May 03, 2022 at 05:42:56PM -0700, Christian Huitema wrote:
> As I said, there appear to be two concerns. One is to protect the local
> server from floods, and, as you say, this is much better done by blocking
> the suspicious traffic ahead of the server. The other is, provide at least a
> modicum of defense against attacks that use the QUIC server as a reflector.
> For example:
> 
> * attacker sends a "carefully crafted" UDP packet to QUIC server, spoofing
> the source address and port of a target,
> 
> * QUIC server processes the packet and produces the expected "response"
> 
> * target attempts to process the packet and something bad happens.
> 
> The concern here is not so much volumetric attacks, which can be mitigated
> with a combination of not amplifying traffic and carefully using retry. We
> are also concerned about "Request Forgery" attacks, in which the "response"
> activates a bug in the target. For example, QUIC sends a long packet to a
> poorly written game server, and the game server crashes when trying to
> process that packet.

Sure but this problem has been existing for as long as UDP services were
reachable on the net, using ECHO, CHARGEN or RPCSS as generators or
reflectors, and a variety of other services as targets.

> There may also be variants of that attack in which an
> attacking server bounces packets through a QUIC client.

This is much less likely to happen since most clients nowadays are behind
stateful NAT, so the attacking server must use the same source address as
an existing connection whose server will be attacked. However, I predict
that we'll see new privacy attacks based on IP ID generation so that an
attacker can try to guess whether or not a client currently has an
established QUIC connection with a given server. The principle will be
the same as with TCP and will consist in trying to forge a packet as
coming from the supposed server and seeing if the client responds/retries,
causing a change in the IP ID generation of subsequent packets. I think
by now most operating systems are safe regarding this, but we'll see in
the long term.

> Clearly, servers that can be crashed by a few magic packets have no business
> being reachable from the Internet. But the concern here is that by spoofing
> the source address, an attacker located outside of the server network can
> reflect and attack towards a vulnerable server inside that network, even if
> firewalls isolate that vulnerable server from the Internet.

This one is already solved by anti-spoofing at the edge and is not
specific to QUIC, and it's valid for other protocols (including ICMP
and TCP). If you do not filter your own sources at the edge, your
internal network is already exposed to attacks and scans. Such attacks
have been widely known and documented for 2 decades now, and similarly
to what you mentioned above, infrastructure that doesn't filter its own
source at the edge has no business being reachable from the net.

But it's very possible that we'll see a reload of such attacks with
QUIC connection attempts from ports 514 (syslog), 162 (snmp trap) or
even 161 (snmp) as an attempt to catch such misconfigured networks.
Again this is another reason for servers not to accept connections
from unprivileged ports.

> > I would suggest that port filtering at the application layer is only
> > used to decide whether to respond or not (i.e. should I send a retry
> > for a packet that parses correctly). For example it could be suggested
> > that packets coming from suspicious but valid source ports ought to
> > be double-checked with a retry to preserve local resources. But that
> > will not be used to save CPU anyway.
> 
> Yes. Makes sense. But there are four ways by which QUIC can bounce packets:
> 
> * Initial packets from the client will cause the server to send a flight of
> up to 3 Handshake packets. The Retry mechanism could be used there.
> 
> * Initial packets from the client with an unsupported version number cause
> the server to send a Version Negotiation packet. These are short packets,
> but up to 256 bytes of the VN packet content can be predetermined by the
> attacker.

256 bytes are sufficient to cause a large DNS or NTP request, so what could
happen is an attacker sending a packet from port 53 or 123 of a server
known for responding large packets, to a victim QUIC server, that will
respond to this server and cause a large response to be sent to it. As
such the attacker will only need to send 256 bytes to the victim, and
the victim will itself request the extra few kB from another location.
This is another form of amplification attack, which is properly addressed
by filtering these invalid source ports.

> * 1RTT packets sent to a server with a unknown Connection ID may cause the
> server to send a Stateless Reset packet. The Stateless Reset packet must be
> shorter than the incoming packet, so there is no amplification. The content
> is hard to predict, but who knows.
> 
> * After a connection has been established and verified, either party can
> send a "path validation" packet from a "new" address and port. Clients or
> servers will respond with a path validation packet of their own, trying to
> reach the new address. The validation attempt could be repeated multiple
> time, typically 3 times. At least 20 bytes at the beginning of the packet
> can be controlled by the attacker, and possibly many more.
> 
> If an application just drops packets from a suspicious port, it mitigates
> all 4 avenues of attack. If it want precision instead, it has to write
> specific code at four locations. RFC 9000 details these attacks, but mostly
> says "protect using ingress filtering, per BCP38".

There *will* inevitably be some problems at the beginning, but one good
thing is that solutions are reasonably simple, can instantly be deployed
in field using port filtering, and will quickly land in the code itself
because the various stack authors have full control of their code.

It could have been great to decide that the first byte in the protocol
should be random. This would have significantly voided most possibilities
of abuse by making the first byte non-predictable nor controllable. But
this is easy to say afterwards and it could come with its own set of
problems (e.g. fingerprinting from the random sequence) ;-)

> > Maintaining a public list of well-known suspicious ports can be both
> > helpful and dangerous. It's helpful in that it shows implementers that
> > some ranges should never appear as they're normally reserved for
> > listening services, that some numbers are well known and widely
> > deployed, and that some are less common but appear anywhere, indicating
> > that a range is not sufficient. This should help design a flexible
> > filtering mechanism. But it can also be dangerous if applied as-is:
> > this list cannot remain static as new entries may have to be added
> > within a few minutes to hours hours and each addition will cause extra
> > breakage, thus some previous ones will likely be dropped after a
> > previous service stopped being widely attacked. I.e. the list in
> > question likely needs to be accessible by configuration in field and
> > not be hard-coded.
> AFAIK, such lists are hard coded in quite a few implementations. Even if
> they are provided in a configuration file, changing the file requires
> restarting the server. So "within a few minutes" may be problematic.

It's not a problem anymore once under heavy attack, believe me ;-)
Usually production management rules are very strict, but once a site is
under DDoS and zero useful traffic passes, all the rules suddenly change
and you don't even need a permission anymore to change a config file and
restart a process multiple times until the site recovers! The worst
moment is when it starts to work again and you would like to clean up
all the mess in your config files but you're not allowed to touch
anything anymore, as if your actions were the ones that caused the
problem in the first place.

> > Other approaches could work fairly well, such as keeping a rate counter
> > of incoming packets per source port (e.g. unparsable or initial packets)
> > which, above a moderate value, would serve to send retry, and above a
> > critical value, can be used to decide to block the port (possibly earlier
> > in the chain). This remains reasonably cheap to implement, though it may
> > end up causing some flapping as the rate will fall after the port is
> > blocked upstream, which may lead to it being reopened before the attack
> > is finished.
> Yes. Should that really be "per source port" or "per source IP + port"?

Per port should be sufficient. There's no reason from the server side that
multiple clients collectively decide to use the same source port to send
legit traffic. And using the port alone will allow to quicky detect a new
internet-wide attack coming from a randomly vulnerable service (IoT
gateways, ToIP etc) regardless of their address. I don't know how this
would be distributed, but I guess that if a source port is responsible
for 100 times more traffic than all other one combined, it will usually
be a good indicator that you don't want to deal with that traffic anymore.

> > I think we'll discover new fun things over time and will learn new
> > attacks and workarounds as deployments grow.
> 
> I have not heard of actual attacks "in the wild", but who knows...

It's too early, stacks are still evolving and the level of deployment
is not high yet. That doesn't mean nobody is trying to elaborate new
attacks. For example yesterday on my home servers which do not advertise
QUIC I received one short packet from port 443 to port 443 from a host
named "scanners.labs.something", and a 1238-byte one from a random port
to 443 from a host named "scan-19a.somethingelse". Thus it shows that
studies are already in progress. We just need to be patient :-)

Regards,
Willy