Re: [TLS] TLS 1.3 / RSA-PSS and unusual key sizes

[Tim Taubert <[e-mail]>]

Hanno Böck wrote:
> Hi,
> 
> I wanted to warn people about a potential source of bugs with the
> deployment of RSA-PSS in TLS 1.3.
> 
> Usually the RSA key modulus is a multiple of 8 (2048, 4096 etc.).
> However there's no rule that RSA keys can't have other sizes.
> 
> Implementing PSS with support for arbitrary key sizes is a bit more
> complicated than implementing it for multiples of 8. I wrote the PSS
> implementation of NSS as a summer of code project a couple of years ago
> and I remember that my first implementation completely failed to
> consider this. (The fix for that never got merged afair, I informed NSS
> developers about this.)

Thanks again Hanno for bringing this to our attention. We patched it
yesterday in NSS 3.29, soon to be merged into Firefox 53.

If anyone else is about to take a look, feel free to use the official
RSA-PSS vectors I converted to SPKI/PKCS8, so you don't have to:

https://hg.mozilla.org/projects/nss/file/c61324648525/gtests/pk11_gtest/pk11_rsapss_vectors.h

- Tim


> Back then I also reported a bug in OpenSSL:
> https://rt.openssl.org/Ticket/Display.html?id=2315&user=guest&pass=guest
> 
> Long story short: It's not unlikely that there are more PSS
> implementations having problems with this.
> So I strongly recommend that all implementors of TLS 1.3 test their
> implementations for key sizes from n*8+1 to N*8+7.
> 
> Such keys are rare, but they do exist in the wild. If implementations
> failing on that get shipped widely we may see random unexplained errors
> when people start migrating to TLS 1.3 in masses.
> 
> I had actually considered proposing to change TLS 1.3 in a way that
> such keys would be simply forbidden. But I did a check on the censys
> data and there were too many of them in the wild, so I thought it
> wasn't a feasible idea.
>