Re: [Asrg] Some statistics on SPF and spam

darxus@chaosreigns.com Tue, 12 February 2013 19:55 UTC

Return-Path: <darxus@chaosreigns.com>
X-Original-To: asrg@ietfa.amsl.com
Delivered-To: asrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6DBAF21F8D49 for <asrg@ietfa.amsl.com>; Tue, 12 Feb 2013 11:55:53 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JvnWaj9Pp0ev for <asrg@ietfa.amsl.com>; Tue, 12 Feb 2013 11:55:52 -0800 (PST)
Received: from panic.chaosreigns.com (panic.chaosreigns.com [IPv6:2600:3c01::f03c:91ff:fe96:340b]) by ietfa.amsl.com (Postfix) with ESMTP id 7A33421F8C51 for <asrg@irtf.org>; Tue, 12 Feb 2013 11:55:49 -0800 (PST)
Received: by panic.chaosreigns.com (Postfix, from userid 1000) id 14583CD336; Tue, 12 Feb 2013 14:55:49 -0500 (EST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=chaosreigns.com; s=mail; t=1360698949; bh=jSO7Uij3Hy2v8Dr66AVX7kbYri3yThzY0H3+KOcbwZ8=; h=Date:From:To:Subject:References:In-Reply-To; b=lYX7D3QDC1xUymL8r3IidmsJZ0CypWhPxpwqJ/bUom3NJFPEfdHNvFfCccWqXf222 1jhA+2i+2zCwzuN6SELr6ZIG1l7gruMtYLQUKFEGeklVXFbD9cF4fbbm0O9KzjPsJM AT7tXpWs/h+hjZrr6Jzodh91ErZ4GUPK03Rf6hHU=
Date: Tue, 12 Feb 2013 14:55:49 -0500
From: darxus@chaosreigns.com
To: asrg@irtf.org
Message-ID: <20130212195549.GB26133@chaosreigns.com>
References: <0D79787962F6AE4B84B2CC41FC957D0B20BBE549@abn-exch1b.green.sophos>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <0D79787962F6AE4B84B2CC41FC957D0B20BBE549@abn-exch1b.green.sophos>
User-Agent: Mutt/1.5.20 (2009-06-14)
Subject: Re: [Asrg] Some statistics on SPF and spam
X-BeenThere: asrg@irtf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: Anti-Spam Research Group - IRTF <asrg@irtf.org>
List-Id: Anti-Spam Research Group - IRTF <asrg.irtf.org>
List-Unsubscribe: <http://www.irtf.org/mailman/options/asrg>, <mailto:asrg-request@irtf.org?subject=unsubscribe>
List-Archive: <http://www.irtf.org/mail-archive/web/asrg>
List-Post: <mailto:asrg@irtf.org>
List-Help: <mailto:asrg-request@irtf.org?subject=help>
List-Subscribe: <http://www.irtf.org/mailman/listinfo/asrg>, <mailto:asrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Feb 2013 19:55:53 -0000

I didn't see the discussion where you promised to produce this, but I think
the problem is how much non-spam also fails SPF.  

>From ruleqa.spamassassin.org/?daterev=20130211-r1444680-n&rule=%2Fspf :

  MSECS    SPAM%     HAM%     S/O    RANK   SCORE  NAME   WHO/AGE
      0   0.0236   0.9635   0.024    0.15    0.00  SPF_FAIL  
      0   0.0383   0.3059   0.111    0.27    0.00  SPF_SOFTFAIL  

Way more non-spam is failing than spam.

Catching spam is easy.  Doing so without excessive false positives is
what's hard.

On 02/12, Martijn Grooten wrote:
> I had promised to produce some stats on SPF and spam.
> 
> Over the Christmas holidays, I sent over 60k spam messages through 21 spam filters in the spam-filter test I run regularly. I checked the SPF status of the messages and measured how many filters failed to block each message.
> 
> Here are the results:
> SPF fail: 3171 emails, on average missed by 0.24 filters (out of 21) with a standard deviation of 0.04.
> SPF pass: 8106 emails, avg 0.93, stddev 0.23
> SPF softfail: 8672 emails, avg 0.45, stddev 0.09
> SPF neutral: 13466 emails, avg 0.34, stddev 0.04
> SPF none: 26938 emails, avg 0.43, stddev 0.06
> 
> A neater table and a graph can be found here: http://www.virusbtn.com/news/2013/02_04.xml
> 
> Now correlation doesn't imply causation and there are good reasons why the relationship here may not causal, but let's for a moment we assume it is.
> 
> This means that if you're a spammer, failing SPF isn't a good idea, while making sure your emails pass SPF means you're more likely to see your messages delivered, but you by no means get a free ride to users' inboxes.
> 
> If you find a 'clever' way to avoid failing SPF by using a domain with no SPF record, there is only a small improvement in your delivery rates.
> 
> Martijn.
> 
> 
> ________________________________
> 
> Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
> Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.
> _______________________________________________
> Asrg mailing list
> Asrg@irtf.org
> http://www.irtf.org/mailman/listinfo/asrg
> 

-- 
"Begin at the beginning and go on till you come to the end; then stop."
- Lewis Carrol, Alice in Wonderland
http://www.ChaosReigns.com