[DNSOP] FIPS 140-3 mode on RHEL 9 and RSA validation of <2048 keys

[Petr Menšík <[e-mail]>]

Hello,

I have sent already a notification about SHA-1 not validated in default
configuration. However that was not end of the story.

A new and even more severe issue has arisen. Our crypto team is
responsible for preparing RHEL 9 for FIPS 140-3 certification. They said
there is legal obligation to stop using all RSA signatures with keys
shorter than 2048 bits. I expect many of you already know especially
Zone Signing Keys often use 1280 or even 1024b keys without issues.

Current RHEL 9 validates such signatures well, but there is a bug [1]
filled to change that and start enforcing longer keys usage only.
Because it would be enforced at crypto library level (openssl), common
DNSSEC validation software would start failing. On quite a lot of
domains. And failing to resolve such names.

I am looking for the best way out of this. Once the above happens
without any changes to used DNS(SEC) software, the only way to keep
working DNS servers working would be either turning off FIPS more or
DNSSEC validation. I know the result seems quite ridiculous, because all
this has been done in order to increase the security. The only
alternative without changes would be disabling all RSA altogether and
providing a long list of ECDSA trust anchors for TLD domains, where at
least ECDSA would offer protection.

I think the only good way would be starting considering shorter keys as
insecure in FIPS mode. Anything above those domains would be without
DNSSEC protection, but at least single root key could be used. Do you
think it would be safe once the DS digest is checked on the key, that
key is then can be marked as insecure instead of bogus. Just as if the
algorithm were unsupported, but after checking the length of key.
Because the crypto library would refuse any operation, including
signature validation, on the given key. I think the library would not
even allow loading that key.

Do you think there might be possible attacks on such algorithm? Would it
be possible to take advantage of it and downgrade also names protected
by long enough keys? Would the behaviour mentioned above require updated
protocol or just modification of existing software? Is the described
behaviour forbidden by existing RFCs?

Any help or comments would be very welcome.

Best Regards,
Petr Menšík

1. https://bugzilla.redhat.com/show_bug.cgi?id=2077884

-- 
Petr Menšík
Software Engineer
Red Hat, http://www.redhat.com/
email: [e-mail]
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB