Re: [Cfrg] Publicly verifiable benchmarks

Michael Hamburg <> Mon, 13 October 2014 17:49 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id AD6441A1AB3 for <>; Mon, 13 Oct 2014 10:49:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 1.641
X-Spam-Level: *
X-Spam-Status: No, score=1.641 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888, HELO_MISMATCH_ORG=0.611, HOST_MISMATCH_NET=0.311, RDNS_DYNAMIC=0.982, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, THIS_AD=0.086] autolearn=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 7JWjKrUMNhuv for <>; Mon, 13 Oct 2014 10:49:54 -0700 (PDT)
Received: from ( []) (using TLSv1.1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 726E41A1BC3 for <>; Mon, 13 Oct 2014 10:49:51 -0700 (PDT)
Received: from [] (unknown []) by (Postfix) with ESMTPSA id 2FDE0F5EAE; Mon, 13 Oct 2014 10:48:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;; s=sldo; t=1413222487; bh=MzC9FiWfou8JK04GK6Yy68OQywkg4XAdJqH80rO2VMY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From; b=kA1vuaAEPiI8GRgAePDZxmPCPZ53Kt64J56TZ/WEthz8KK6HSXTd9ozREb+j5p8zm 0cRTUmCw52h9xg+JIaAdGD+pugOd3V2qOATnqe2OPSQH9bqzV+JWVobaSw0hVl7kC5 oi3dKhrGzKZHFP0EgHcRYeOB7HVwem/9GAu+6gAM=
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\))
From: Michael Hamburg <>
In-Reply-To: <>
Date: Mon, 13 Oct 2014 10:49:49 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <>
To: "D. J. Bernstein" <>
X-Mailer: Apple Mail (2.1990.1)
Subject: Re: [Cfrg] Publicly verifiable benchmarks
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Crypto Forum Research Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 13 Oct 2014 17:49:55 -0000

> On Oct 13, 2014, at 4:36 AM, D. J. Bernstein <> wrote:

> David Jacobson writes:
>> It would be nice if you tagged implementations (of algorithms where it
>> matters) into according to leakage resistance.
> Yes. Some implementors advertise constant-time software, but right now
> SUPERCOP doesn't provide a structured mechanism for this advertisement.
> One of the difficulties here is that some people are more stringent than
> others in what they mean by "constant time"; I'll write a separate
> message about this.

You used to have int timingattacks(void) { … }.  Or was that just a BATMAN thing?

> Michael Hamburg writes:
>> some of the machines on have quirks
>> (turboboost, mismatched cycle counter frequency, etc) which can make
>> the data difficult to interpret and reproduce.
> Turbo Boost is noted as "boost" in red (as is Turbo Core), and is also
> easy to spot as unusually large gaps between the quartiles. See, e.g.,
> The SUPERCOP documentation has a "Reducing randomness in benchmarks"
> section that tells people how to turn off hyperthreading and Turbo
> Boost. Eventually we'd like to measure what the actual Turbo Boost
> speedup is, but this isn't as easy as it sounds.

It sure doesn’t sound easy.

> Some CPUs don't give the OS full control over clock frequencies. Of
> course, clock frequency makes far less of a difference in cycle counts
> than it makes in other metrics such as operations per second, but it
> does sometimes make a noticeable difference, especially for code that
> doesn't fit into L1 cache.

At least on Intel CPUs, the counter read by RDTSC runs at a constant rate which may or may not have anything to do with the CPU’s actual frequency.  So for example, on a Haswell MacBook Air 1.7GHz (boost to 3.3GHz), the counter runs at 2.2GHz even when TurboBoost is disabled.  This also means that unless you have privilege to read the pcr cycle counter, boosting does ruin the cycle counts just as much as the ops/second.

In the past I’ve seen data on which looked like it may have been affected by this problem: it showed reasonable quartiles but noticeably faster or slower speeds across all benchmarks.  But that was some time ago, so maybe it’s gone now.

>  Of course, asm code produces much less variability.

That’s a good point.  The downside is that you have to target a specific architecture (eg choose for each directory what vector support you have), but that’s no different from what most library packagers do.

— Mike