[Asrg] Some statistics on SPF and spam

Martijn Grooten <martijn.grooten@virusbtn.com> Tue, 12 February 2013 11:01 UTC

Return-Path: <martijn.grooten@virusbtn.com>
X-Original-To: asrg@ietfa.amsl.com
Delivered-To: asrg@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id B169521F8CD0 for <asrg@ietfa.amsl.com>; Tue, 12 Feb 2013 03:01:21 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -8.092
X-Spam-Status: No, score=-8.092 tagged_above=-999 required=5 tests=[AWL=-0.093, BAYES_50=0.001, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id rGASfQZjr0W6 for <asrg@ietfa.amsl.com>; Tue, 12 Feb 2013 03:01:21 -0800 (PST)
Received: from mx6.sophos.com (mx6.sophos.com []) by ietfa.amsl.com (Postfix) with ESMTP id D749B21F8CD1 for <asrg@irtf.org>; Tue, 12 Feb 2013 03:01:19 -0800 (PST)
Received: from mx6.sophos.com (localhost.localdomain []) by localhost (Postfix) with SMTP id 3C1AF7518DF for <asrg@irtf.org>; Tue, 12 Feb 2013 11:01:17 +0000 (GMT)
Received: from ABN-EXCH1A.green.sophos (unknown []) by mx6.sophos.com (Postfix) with ESMTPS id 174727518D6 for <asrg@irtf.org>; Tue, 12 Feb 2013 11:01:17 +0000 (GMT)
Received: from abn-exch1b.green.sophos ([fe80::dc96:facf:3d2c:c352]) by ABN-EXCH1A.green.sophos ([fe80::67:3150:dacd:910d%16]) with mapi id 14.02.0247.003; Tue, 12 Feb 2013 11:01:17 +0000
From: Martijn Grooten <martijn.grooten@virusbtn.com>
To: "Anti-Spam Research Group - IRTF (asrg@irtf.org)" <asrg@irtf.org>
Thread-Topic: Some statistics on SPF and spam
Thread-Index: Ac4JDkhL3olESDoZQtiqFxWmWQ01Cg==
Date: Tue, 12 Feb 2013 11:01:16 +0000
Message-ID: <0D79787962F6AE4B84B2CC41FC957D0B20BBE549@abn-exch1b.green.sophos>
Accept-Language: en-GB, en-US
Content-Language: en-US
x-originating-ip: []
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: [Asrg] Some statistics on SPF and spam
X-BeenThere: asrg@irtf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: Anti-Spam Research Group - IRTF <asrg@irtf.org>
List-Id: Anti-Spam Research Group - IRTF <asrg.irtf.org>
List-Unsubscribe: <http://www.irtf.org/mailman/options/asrg>, <mailto:asrg-request@irtf.org?subject=unsubscribe>
List-Archive: <http://www.irtf.org/mail-archive/web/asrg>
List-Post: <mailto:asrg@irtf.org>
List-Help: <mailto:asrg-request@irtf.org?subject=help>
List-Subscribe: <http://www.irtf.org/mailman/listinfo/asrg>, <mailto:asrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Feb 2013 11:01:21 -0000

I had promised to produce some stats on SPF and spam.

Over the Christmas holidays, I sent over 60k spam messages through 21 spam filters in the spam-filter test I run regularly. I checked the SPF status of the messages and measured how many filters failed to block each message.

Here are the results:
SPF fail: 3171 emails, on average missed by 0.24 filters (out of 21) with a standard deviation of 0.04.
SPF pass: 8106 emails, avg 0.93, stddev 0.23
SPF softfail: 8672 emails, avg 0.45, stddev 0.09
SPF neutral: 13466 emails, avg 0.34, stddev 0.04
SPF none: 26938 emails, avg 0.43, stddev 0.06

A neater table and a graph can be found here: http://www.virusbtn.com/news/2013/02_04.xml

Now correlation doesn't imply causation and there are good reasons why the relationship here may not causal, but let's for a moment we assume it is.

This means that if you're a spammer, failing SPF isn't a good idea, while making sure your emails pass SPF means you're more likely to see your messages delivered, but you by no means get a free ride to users' inboxes.

If you find a 'clever' way to avoid failing SPF by using a domain with no SPF record, there is only a small improvement in your delivery rates.



Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.