Re: [Trans] What's the load on a CT log?

Ben Laurie <benl@google.com> Thu, 13 March 2014 21:48 UTC

Return-Path: <benl@google.com>
X-Original-To: trans@ietfa.amsl.com
Delivered-To: trans@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 74F3A1A0778 for <trans@ietfa.amsl.com>; Thu, 13 Mar 2014 14:48:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.926
X-Spam-Level:
X-Spam-Status: No, score=-1.926 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, RP_MATCHES_RCVD=-0.547, SPF_PASS=-0.001] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fQ1C3IYU-h1N for <trans@ietfa.amsl.com>; Thu, 13 Mar 2014 14:48:12 -0700 (PDT)
Received: from mail-vc0-x22b.google.com (mail-vc0-x22b.google.com [IPv6:2607:f8b0:400c:c03::22b]) by ietfa.amsl.com (Postfix) with ESMTP id 555191A0A39 for <trans@ietf.org>; Thu, 13 Mar 2014 14:48:12 -0700 (PDT)
Received: by mail-vc0-f171.google.com with SMTP id lg15so1819131vcb.16 for <trans@ietf.org>; Thu, 13 Mar 2014 14:48:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=D7gr2ZgoWcm1IZcJ5Vr+mUOp2WATe33hiw3846R1PuQ=; b=P95vgRV0zyxSYp87MarKjjZRhnVyavrhjyfTpgkbhQYGDfAwf4KY9fF2d0ItimQ28x MsuuEw7zWYH8GwYURxCEBtGO5H5WH9r7/RJKmwXqYBxdUdGyCLfjPMJwtgce39rxSV6v aMK/DY7OXLJrD9NSWBTwj4c32ItVXvmpQXYa1oCsjF0q4kg4psBEaIkRdkD9BtC0Qj+q Qv2q7rG3cXiqy7EdcRorjxpTZ0q6ModtToiVCRdRYa0E7i3QGJn7aOSVtsP1DAZOMyeT DBF8yPLDLdlvtIvI/roaNz96TVpBcd5hOJeq1/MXGrqt7CRkzrkjyNapM650s1MGP3Lj lbbg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=D7gr2ZgoWcm1IZcJ5Vr+mUOp2WATe33hiw3846R1PuQ=; b=d+JCbXvFwfH6KWqQT58/qMXUELQU5yrUlIQfmvg85Jhv7eAjmHPXqFET/eeQAT6lCy gbIYtvgO163Xufv1OZKa0st37zhKY7Y3e1Rar5BVyk8cN61ifV9PaK3AQwt+8aPVJCx3 PB+BebZSLEQ2cNO29/dvS8XM3scEDHHyEtpdEMpXdExKMKeTbkXhUyJ44rzWtVNbOHwB YM+piLEV4AayDVporrF16HNyelNziTekg4meIW7G5kAIvKVtVYBda4p7RBchZ2UUmnsc ZwcQvKfWIsXgu/3v0f1meBaY2UzU4f9Y9E3pwodEu7OCt24BzUHw+ZuJ2PjKdzQLlTUx qbeg==
X-Gm-Message-State: ALoCoQm17qoK2Xg3TB7AP3LBpXAyg0oQ7lULiSItitCELIr33gvr/YFBGMZCBeakzhFkZubfegC+JEJpNkDzIs1rqHFW+OCoDvhRCl5PVkVHiLxihD334vJrrFcm0ZTovbacyJROluLon9Ex1YPHegXIwoKph+LsGB57yVhGlVgwCbqaw9CIVKnukVMAh3v9e3PwA0V+LJNx
MIME-Version: 1.0
X-Received: by 10.220.106.84 with SMTP id w20mr3284221vco.18.1394747285463; Thu, 13 Mar 2014 14:48:05 -0700 (PDT)
Received: by 10.52.230.105 with HTTP; Thu, 13 Mar 2014 14:48:05 -0700 (PDT)
In-Reply-To: <53221499.40301@comodo.com>
References: <CABrd9SR4G6hEUEW9yHLyS40Km3+jmK8K-tEjLMjLqN1M+Go_=g@mail.gmail.com> <53221499.40301@comodo.com>
Date: Thu, 13 Mar 2014 21:48:05 +0000
Message-ID: <CABrd9SSdZZRGr2Q6CoHsquNM-TOFSJehjEACPXzEdK=h7=CpAg@mail.gmail.com>
From: Ben Laurie <benl@google.com>
To: Rob Stradling <rob.stradling@comodo.com>
Content-Type: text/plain; charset="UTF-8"
Archived-At: http://mailarchive.ietf.org/arch/msg/trans/RWCtpnbk7v9JumO7kjhKU6leTcI
Cc: "therightkey@ietf.org" <therightkey@ietf.org>, "trans@ietf.org" <trans@ietf.org>, "certificate-transparency@googlegroups.com" <certificate-transparency@googlegroups.com>, CABFPub <public@cabforum.org>
Subject: Re: [Trans] What's the load on a CT log?
X-BeenThere: trans@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Public Notary Transparency working group discussion list <trans.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/trans>, <mailto:trans-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/trans/>
List-Post: <mailto:trans@ietf.org>
List-Help: <mailto:trans-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/trans>, <mailto:trans-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Mar 2014 21:48:14 -0000

On 13 March 2014 20:27, Rob Stradling <rob.stradling@comodo.com> wrote:
> I'm not sure average load tells the whole story.

Agreed.

> Won't there be a surge in audit traffic in the aftermath of a busy site
> installing a new cert?

Yes.

>
>
> On 13/03/14 16:06, Ben Laurie wrote:
>>
>> Several people have asked me this recently. Here's a nice way to estimate
>> load.
>>
>> Let's assume a single log that takes all the load.
>>
>> Firstly, we see about 5,000 new certificates a day, so that's around
>> 0.06 new certificates per second. Clearly a trivial load.
>>
>> Next is load from audit (i.e. from browsers that wish to validate SCTs
>> accompanying certificates they see). Given some assumptions, we can
>> calculate the load from audit.
>>
>> * Clients cache audit results.
>>
>> * There are approximately b = 2.5B browsers in the world
>> (http://www.internetworldstats.com/stats.htm).
>>
>> * The average user visits w = 89 websites a month
>> (http://www.creditloan.com/blog/how-the-world-spends-its-time-online/
>> quoting a Nielsen report). Assume these are all TLS sites.
>>
>> * Assume a certificate lifetime of l = 12 months.
>>
>> So, each user sees w / l new certificates a month. Each new
>> certificate needs to be audited, which means in practice, three web
>> operations (fetch STH, fetch STH consistency proof, fetch SCT
>> inclusion proof) - it might be a good idea to create a new API to do
>> all three in one go.
>>
>> So, total average load is 3 * b * w / l ~ 20,000 web fetches per
>> second. If we optimise the API we can get that down to 7,000 qps. Each
>> query (in the optimised case) would be around 3 kB, which gives a
>> bandwidth of around 150 kb/s.
>>
>> Monitors add extra load, but should only be at around the new
>> certificate rate - i.e. ~ .06 * number of monitors fetches per second.
>>
>> IMO, this is achievable on a single machine (modulo reliability), with
>> some care. Clearly not a vast farm, however its done.
>>
>> In practice, no one log would have to take this full load, this is a
>> worst case analysis.
>>
>> _______________________________________________
>> Trans mailing list
>> Trans@ietf.org
>> https://www.ietf.org/mailman/listinfo/trans
>>
>
> --
> Rob Stradling
> Senior Research & Development Scientist
> COMODO - Creating Trust Online
> Office Tel: +44.(0)1274.730505
> Office Fax: +44.(0)1274.730909
> www.comodo.com
>
> COMODO CA Limited, Registered in England No. 04058690
> Registered Office:
>   3rd Floor, 26 Office Village, Exchange Quay,
>   Trafford Road, Salford, Manchester M5 3EQ
>
> This e-mail and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the sender by
> replying to the e-mail containing this attachment. Replies to this email may
> be monitored by COMODO for operational or business reasons. Whilst every
> endeavour is taken to ensure that e-mails are free from viruses, no
> liability can be accepted and the recipient is requested to use their own
> virus checking software.