Re: [TLS] TLS 1.2 - is it allowed to strip the leading zero byte(s) in RSA signature in ServerKeyExchange?

The signature is invalid. The client is correct to reject it, and the
server is incorrect to produce it.

RFC5246 cites PKCS1 (then RFC3447, now RFC8017). Both versions spell out
the signing and verifying operations explicitly. The signing operation must
produce a fixed-width output and the verification operation must reject
incorrectly-sized inputs:
https://tools.ietf.org/html/rfc3447#section-8.2.1
https://tools.ietf.org/html/rfc3447#section-8.2.2
https://tools.ietf.org/html/rfc8017#section-8.2.1
https://tools.ietf.org/html/rfc8017#section-8.2.2

On Wed, Feb 12, 2020 at 1:27 AM M K Saravanan <mksarav@gmail.com> wrote:

> Hi,
>
> I recently encountered the below issue:
>
> TLS1.2
> ECDHE_RSA
> server certificate: 2048-bit RSA (= 256 bytes)
> ServerKeyExchange hash/sign algorithm: rsa_pkcs1_sha1
>
> The server was sending the ServerKeyExchange with 255 byte as length for
> the RSA signature (i.e. the leading zero was stripped) instead of 256 like
> this:
>
> ====================
> Handshake Protocol: Server Key Exchange
>     Handshake Type: Server Key Exchange (12)
>     Length: 328
>     EC Diffie-Hellman Server Params
>         Curve Type: named_curve (0x03)
>         Named Curve: secp256r1 (0x0017)
>         Pubkey Length: 65
>         Pubkey: 042206562efea8bd47bf014a9e650c42f27078643c553671…
>         Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
>         Signature Length: 255
>         Signature: d1bf915eca2ec0bcdda6f90a398fe5378d2028a22574d213…
> ====================
>
> Is this allowed?  i.e. stripping the leading zero of the RSA signature and
> marking the length as 255?   It is not clear to me from the RFC5246 whether
> it is allowed or not.
>
> (client was failing to verify the signature due to this).
>
> with regards,
> Saravanan
>
> _______________________________________________
> TLS mailing list
> TLS@ietf.org
> https://www.ietf.org/mailman/listinfo/tls
>