Re: [TLS] TLS 1.2 - is it allowed to strip the leading zero byte(s) in RSA signature in ServerKeyExchange?

Peter Gutmann <pgut001@cs.auckland.ac.nz> Wed, 12 February 2020 16:10 UTC

Return-Path: <pgut001@cs.auckland.ac.nz>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 20071120104 for <tls@ietfa.amsl.com>; Wed, 12 Feb 2020 08:10:05 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.198
X-Spam-Level:
X-Spam-Status: No, score=-4.198 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=auckland.ac.nz
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wyrn2jX7ShzG for <tls@ietfa.amsl.com>; Wed, 12 Feb 2020 08:10:04 -0800 (PST)
Received: from mx4-int.auckland.ac.nz (mx4-int.auckland.ac.nz [130.216.125.246]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7470F1200FB for <tls@ietf.org>; Wed, 12 Feb 2020 08:10:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=auckland.ac.nz; i=@auckland.ac.nz; q=dns/txt; s=mail; t=1581523804; x=1613059804; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=f0oN8aBPWVw+lITKd7JqlXI63IC0I7CUjAtmzcL35sc=; b=LgaLf0G/WWp3OyhIK1neA7tIw20HyqAZwMDtgQsjzBJHUsDn1gKUGPXr /DV/m83FHfKJSjeqdgMT916MQ9fIPL0ZuyLYXUSa79rcVEYn8WNIaR9bK k6fzqoeO53b6ZFHpoNvUSAOYQkK5e4RdzByDht/bxG2jcMTC8do7um67k Z/+trepnsgcXrqC0oZAAe6ZIrSlhyP+JAIwO559NtgZhzHwN9/rcD+Huq JHznn+ae15I9CQFDkaYquGoqzvlkzJg8v64S+TS1Vbe1yk7USJoQTYNOT yCKDxImq7GZR7aNHmXuNuVPtiwZuFItOYoCifx/yd0VbOVFl3Da6IJrtK A==;
X-IronPort-AV: E=Sophos;i="5.70,433,1574074800"; d="scan'208";a="114813472"
X-Ironport-HAT: MAIL-SERVERS - $RELAYED
X-Ironport-Source: 10.6.3.9 - Outgoing - Outgoing
Received: from uxcn13-tdc-e.uoa.auckland.ac.nz ([10.6.3.9]) by mx4-int.auckland.ac.nz with ESMTP/TLS/AES256-SHA; 13 Feb 2020 05:10:02 +1300
Received: from uxcn13-ogg-d.UoA.auckland.ac.nz (10.6.2.5) by uxcn13-tdc-e.UoA.auckland.ac.nz (10.6.3.9) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Thu, 13 Feb 2020 05:10:01 +1300
Received: from uxcn13-ogg-d.UoA.auckland.ac.nz ([10.6.2.5]) by uxcn13-ogg-d.UoA.auckland.ac.nz ([10.6.2.5]) with mapi id 15.00.1395.000; Thu, 13 Feb 2020 05:10:01 +1300
From: Peter Gutmann <pgut001@cs.auckland.ac.nz>
To: M K Saravanan <mksarav@gmail.com>, "<tls@ietf.org>" <tls@ietf.org>
Thread-Topic: [TLS] TLS 1.2 - is it allowed to strip the leading zero byte(s) in RSA signature in ServerKeyExchange?
Thread-Index: AQHV4W2CYeXLZ0I+bUablesS8zu9G6gXusN0
Date: Wed, 12 Feb 2020 16:10:00 +0000
Message-ID: <1581523805582.16466@cs.auckland.ac.nz>
References: <CAG5P2e_yELKn_ypt2cVAHtoBeNVrpKLqLwuZprq0bm=h4odrHA@mail.gmail.com>
In-Reply-To: <CAG5P2e_yELKn_ypt2cVAHtoBeNVrpKLqLwuZprq0bm=h4odrHA@mail.gmail.com>
Accept-Language: en-NZ, en-GB, en-US
Content-Language: en-NZ
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [130.216.158.4]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/LSELSI6cxifL0yD4NwtdUNHSaKI>
Subject: Re: [TLS] TLS 1.2 - is it allowed to strip the leading zero byte(s) in RSA signature in ServerKeyExchange?
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tls/>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Feb 2020 16:10:05 -0000

M K Saravanan <mksarav@gmail.com> writes:

>Is this allowed?  i.e. stripping the leading zero of the RSA signature and
>marking the length as 255?   It is not clear to me from the RFC5246 whether
>it is allowed or not.

It's not allowed according to the spec but a number of implementations do it
because their underlying bignum libraries perform leading-zero truncation, so
you're better off allowing it to avoid breakage.

Peter.