[TLS] add challenge in TLS v1.3 to prevent DDOS attack?

["Bingzheng Wu" <[e-mail]>]

Hi all,

TLS is susceptible to DDOS attack.
In TLS v1.3, attackers can generate ClientHello and ClientKeyshare messages easily,
while the server has to consume large amounts of CPU doing asymmetric crypto operations to generate ServerKeyshare and ServerCertificateVerify messages.

So, could we add a challenge-response mode in TLS v1.3 to increase the attacker's cost ?

The mode is disable in usual.
If the server think it's under attack (e.g. >1000 qps), it could enable this mode by responding a HelloRetryRequest (or a new type) message to the client with a challenge.
The client receiving the challenge must solve the challenge which is expensive in CPU, and continue the handshake by carrying the challenge's answer.

For example, server could make the challenge by encrypting an random number by a RSA public key (which is short), and send the private key (which is long) and cipher text to client.

This challenge mode may bring extra CPU consumption for normal clients when the server is under attack, but it can reduce the attack's influence for server, to prevent DDOS attack.

Obviously, this mode must be a part of TLS's main body, while being an extension dose not work.


Is this a feasible idea?

Regards,
Bingzheng Wu