[TLS] Pre-master secret formatting for master secret calculation with DH in TLS 1.x

Andrey Jivsov <openpgp@brainhub.org> Fri, 08 February 2013 19:13 UTC

Return-Path: <openpgp@brainhub.org>
X-Original-To: tls@ietfa.amsl.com
Delivered-To: tls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2C44A21F8B13 for <tls@ietfa.amsl.com>; Fri, 8 Feb 2013 11:13:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.437
X-Spam-Level:
X-Spam-Status: No, score=-0.437 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_NET=0.611, RDNS_NONE=0.1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qu12stAUDRjq for <tls@ietfa.amsl.com>; Fri, 8 Feb 2013 11:13:03 -0800 (PST)
Received: from qmta03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:32]) by ietfa.amsl.com (Postfix) with ESMTP id 4ECD221F8B81 for <tls@ietf.org>; Fri, 8 Feb 2013 11:12:59 -0800 (PST)
Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43]) by qmta03.emeryville.ca.mail.comcast.net with comcast id xpgc1k0050vp7WLA3vCz1f; Fri, 08 Feb 2013 19:12:59 +0000
Received: from [127.0.0.1] ([69.181.162.123]) by omta05.emeryville.ca.mail.comcast.net with comcast id xvCx1k00Y2g33ZR8RvCyxw; Fri, 08 Feb 2013 19:12:58 +0000
Message-ID: <51154DCF.4090407@brainhub.org>
Date: Fri, 08 Feb 2013 11:11:11 -0800
From: Andrey Jivsov <openpgp@brainhub.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120911 Thunderbird/15.0.1
MIME-Version: 1.0
To: tls@ietf.org
Content-Type: text/plain; charset="ISO-8859-1"; format="flowed"
Content-Transfer-Encoding: 7bit
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1360350779; bh=98YDsimmkF/jE7umkr0LotBgzc/rP8zALpiWBOtuPrM=; h=Received:Received:Message-ID:Date:From:MIME-Version:To:Subject: Content-Type; b=lK0632RPFyGJeBx/ml8d8zSOx2KPUSXP/v7Rhg2qTQgvCCMUKFY1Dldg7a2dBxL/k 1h3n76Rbkyj6jnmavEHRIquIVo4po+X6+DHEfptC8nfAeCtFE1ltqVQ/NW82ywgdVe Ay6IyqCajguzqraXVZXrmvqVTbT+9759lWJb9sN0La7ImliLevZot6rP7TXgrcYJMB QgepqGesxm/6P7cVIVZ3oUJSklt0Laft0lcx7rfg+6IHjW4FzRcvqK9rU/x9VcdnAk CDunsflOoM3KBuyKHBSrjkGw1qE417CFs0yhbCA7avoiXmCXvsCmtSBtwDUU3xjXOU 4fpKN1QRskxdA==
X-Mailman-Approved-At: Fri, 08 Feb 2013 11:17:08 -0800
Subject: [TLS] Pre-master secret formatting for master secret calculation with DH in TLS 1.x
X-BeenThere: tls@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "This is the mailing list for the Transport Layer Security working group of the IETF." <tls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tls>, <mailto:tls-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tls>
List-Post: <mailto:tls@ietf.org>
List-Help: <mailto:tls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tls>, <mailto:tls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Feb 2013 19:13:04 -0000

I ran into an interoperability problem between our TLS implementation 
and another very popular one.

The question is what exactly is the formatting  for the RFC224 sec. 
8.1.2 "Diffie-Hellman" shared secret 'Z' ?

The problem arises when the most significant byte of the Z is zero. 
Let's say that the field prime's size is 256. There are two possibilities:

1. export the Z in the big endian format, padding it to 256 (first byte 
in the buffer is zero)
2. export the Z in the big endian format, placing the result in a 255 
byte buffer.

I would say that this should be #1, because it is the natural 
representation of the Z. If there is a language specifying the canonical 
encoding for the Z, I would appreciate the reference.

( Apparently, this problem doesn't clearly manifest itself in real 
deployments because ~ the 1/256 failure rate is masked by 
re-negotiations at the higher protocol level. )

Thank you.