Re: [Trans] What's the load on a CT log?

Rob Stradling <rob.stradling@comodo.com> Thu, 13 March 2014 20:27 UTC

Return-Path: <rob.stradling@comodo.com>
X-Original-To: trans@ietfa.amsl.com
Delivered-To: trans@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 38A991A0A2F for <trans@ietfa.amsl.com>; Thu, 13 Mar 2014 13:27:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.29
X-Spam-Level:
X-Spam-Status: No, score=-1.29 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_MISMATCH_NET=0.611, SPF_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id urdXVtxoNtFz for <trans@ietfa.amsl.com>; Thu, 13 Mar 2014 13:27:13 -0700 (PDT)
Received: from ian.brad.office.comodo.net (eth5.brad-fw.brad.office.ccanet.co.uk [178.255.87.226]) by ietfa.amsl.com (Postfix) with ESMTP id 1A6561A0473 for <trans@ietf.org>; Thu, 13 Mar 2014 13:27:12 -0700 (PDT)
Received: (qmail 28465 invoked by uid 1000); 13 Mar 2014 20:27:06 -0000
Received: from nigel.brad.office.comodo.net (HELO [192.168.0.58]) (192.168.0.58) (smtp-auth username rob, mechanism plain) by ian.brad.office.comodo.net (qpsmtpd/0.40) with (AES128-SHA encrypted) ESMTPSA; Thu, 13 Mar 2014 20:27:06 +0000
Message-ID: <53221499.40301@comodo.com>
Date: Thu, 13 Mar 2014 20:27:05 +0000
From: Rob Stradling <rob.stradling@comodo.com>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Thunderbird/24.3.0
MIME-Version: 1.0
To: Ben Laurie <benl@google.com>, "trans@ietf.org" <trans@ietf.org>, "therightkey@ietf.org" <therightkey@ietf.org>, "certificate-transparency@googlegroups.com" <certificate-transparency@googlegroups.com>, CABFPub <public@cabforum.org>
References: <CABrd9SR4G6hEUEW9yHLyS40Km3+jmK8K-tEjLMjLqN1M+Go_=g@mail.gmail.com>
In-Reply-To: <CABrd9SR4G6hEUEW9yHLyS40Km3+jmK8K-tEjLMjLqN1M+Go_=g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: http://mailarchive.ietf.org/arch/msg/trans/w8J34aIPWaW58lHHngrqRlMbCDw
Subject: Re: [Trans] What's the load on a CT log?
X-BeenThere: trans@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Public Notary Transparency working group discussion list <trans.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/trans>, <mailto:trans-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/trans/>
List-Post: <mailto:trans@ietf.org>
List-Help: <mailto:trans-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/trans>, <mailto:trans-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Mar 2014 20:27:17 -0000

I'm not sure average load tells the whole story.

Won't there be a surge in audit traffic in the aftermath of a busy site 
installing a new cert?

On 13/03/14 16:06, Ben Laurie wrote:
> Several people have asked me this recently. Here's a nice way to estimate load.
>
> Let's assume a single log that takes all the load.
>
> Firstly, we see about 5,000 new certificates a day, so that's around
> 0.06 new certificates per second. Clearly a trivial load.
>
> Next is load from audit (i.e. from browsers that wish to validate SCTs
> accompanying certificates they see). Given some assumptions, we can
> calculate the load from audit.
>
> * Clients cache audit results.
>
> * There are approximately b = 2.5B browsers in the world
> (http://www.internetworldstats.com/stats.htm).
>
> * The average user visits w = 89 websites a month
> (http://www.creditloan.com/blog/how-the-world-spends-its-time-online/
> quoting a Nielsen report). Assume these are all TLS sites.
>
> * Assume a certificate lifetime of l = 12 months.
>
> So, each user sees w / l new certificates a month. Each new
> certificate needs to be audited, which means in practice, three web
> operations (fetch STH, fetch STH consistency proof, fetch SCT
> inclusion proof) - it might be a good idea to create a new API to do
> all three in one go.
>
> So, total average load is 3 * b * w / l ~ 20,000 web fetches per
> second. If we optimise the API we can get that down to 7,000 qps. Each
> query (in the optimised case) would be around 3 kB, which gives a
> bandwidth of around 150 kb/s.
>
> Monitors add extra load, but should only be at around the new
> certificate rate - i.e. ~ .06 * number of monitors fetches per second.
>
> IMO, this is achievable on a single machine (modulo reliability), with
> some care. Clearly not a vast farm, however its done.
>
> In practice, no one log would have to take this full load, this is a
> worst case analysis.
>
> _______________________________________________
> Trans mailing list
> Trans@ietf.org
> https://www.ietf.org/mailman/listinfo/trans
>

-- 
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online
Office Tel: +44.(0)1274.730505
Office Fax: +44.(0)1274.730909
www.comodo.com

COMODO CA Limited, Registered in England No. 04058690
Registered Office:
   3rd Floor, 26 Office Village, Exchange Quay,
   Trafford Road, Salford, Manchester M5 3EQ

This e-mail and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they are 
addressed.  If you have received this email in error please notify the 
sender by replying to the e-mail containing this attachment. Replies to 
this email may be monitored by COMODO for operational or business 
reasons. Whilst every endeavour is taken to ensure that e-mails are free 
from viruses, no liability can be accepted and the recipient is 
requested to use their own virus checking software.