Re: [spfbis] Review of draft-ietf-spfbis-experiment-05

"Murray S. Kucherawy" <msk@cloudmark.com> Tue, 24 April 2012 19:16 UTC

Return-Path: <msk@cloudmark.com>
X-Original-To: spfbis@ietfa.amsl.com
Delivered-To: spfbis@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 68C2421E80A4 for <spfbis@ietfa.amsl.com>; Tue, 24 Apr 2012 12:16:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.549
X-Spam-Level:
X-Spam-Status: No, score=-102.549 tagged_above=-999 required=5 tests=[AWL=0.050, BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FylOCU0Wjt49 for <spfbis@ietfa.amsl.com>; Tue, 24 Apr 2012 12:16:13 -0700 (PDT)
Received: from mail.cloudmark.com (cmgw1.cloudmark.com [208.83.136.25]) by ietfa.amsl.com (Postfix) with ESMTP id 9F50C21E8097 for <spfbis@ietf.org>; Tue, 24 Apr 2012 12:16:12 -0700 (PDT)
Received: from ht1-outbound.cloudmark.com ([72.5.239.25]) by mail.cloudmark.com with bizsmtp id 1vGZ1j0020ZaKgw01vGZHT; Tue, 24 Apr 2012 12:16:33 -0700
X-CMAE-Match: 0
X-CMAE-Score: 0.00
X-CMAE-Analysis: v=2.0 cv=fNu7LOme c=1 sm=1 a=LdFkGDrDWH2mcjCZERnC4w==:17 a=LvckAehuu68A:10 a=w0_tcEhzsP4A:10 a=zutiEJmiVI4A:10 a=kj9zAlcOel0A:10 a=xqWC_Br6kY4A:10 a=48vgC7mUAAAA:8 a=XplqSab69mvFeSkpBs0A:9 a=5HqY9-NOj3gnySuCbLYA:7 a=CjuIK1q_8ugA:10 a=lZB815dzVvQA:10 a=l3v_lM9MB44WcsQu:21 a=DgUcOdrRW9gkQR-3:21 a=LdFkGDrDWH2mcjCZERnC4w==:117
Received: from EXCH-MBX901.corp.cloudmark.com ([fe80::addf:849a:f71c:4a82]) by exch-htcas901.corp.cloudmark.com ([fe80::2524:76b6:a865:539c%10]) with mapi id 14.01.0355.002; Tue, 24 Apr 2012 12:16:11 -0700
From: "Murray S. Kucherawy" <msk@cloudmark.com>
To: "spfbis@ietf.org" <spfbis@ietf.org>
Thread-Topic: [spfbis] Review of draft-ietf-spfbis-experiment-05
Thread-Index: AQHNIMqwFmLLWp0VJEuOTXJcBTjdz5anwbsggAETc4D//6ThEIAAgQKAgAFgppA=
Date: Tue, 24 Apr 2012 19:16:11 +0000
Message-ID: <9452079D1A51524AA5749AD23E003928101397@exch-mbx901.corp.cloudmark.com>
References: <CAC4RtVAV5PH+VMzppVxAQgGq0f28ARN846e17G_8sbLCThm-KA@mail.gmail.com> <9452079D1A51524AA5749AD23E0039280FED0D@exch-mbx901.corp.cloudmark.com> <CAJ4XoYf2KNLsqzrrM39bWo1Z1Fun1qEiNMYstLf2ZCaaUDSzmA@mail.gmail.com> <9452079D1A51524AA5749AD23E0039280FF5C4@exch-mbx901.corp.cloudmark.com> <CAJ4XoYe1Vkge=2iWrFgzRyZL-XVt-7bhUCf=xJHhvZcR6mGFiA@mail.gmail.com>
In-Reply-To: <CAJ4XoYe1Vkge=2iWrFgzRyZL-XVt-7bhUCf=xJHhvZcR6mGFiA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.20.2.121]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudmark.com; s=default; t=1335294993; bh=HpIkHTNu7t0uHW4NFkMpVvbRmCbcMrftCcXXuEW+G7Y=; h=From:To:Subject:Date:Message-ID:References:In-Reply-To: Content-Type:Content-Transfer-Encoding:MIME-Version; b=HcTxP46CFbx+Cht+7+4RKx6JWu0gAPUyxlTYCAlkmcS0zwRl/JG0SvmWY6J6oUqC4 9zM2E4HvwWv34IeZaNYD9Lk8oqvwJzFRZNz2TW/o3SrhTw1Rb4OdqU0vcz4PU4FIVg 6R7RD4Uk1kMxWDGBftreLwpfnllfATsKUyBHXtlQ=
Subject: Re: [spfbis] Review of draft-ietf-spfbis-experiment-05
X-BeenThere: spfbis@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: SPFbis discussion list <spfbis.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/spfbis>, <mailto:spfbis-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/spfbis>
List-Post: <mailto:spfbis@ietf.org>
List-Help: <mailto:spfbis-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/spfbis>, <mailto:spfbis-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 19:16:14 -0000

[resending; original appears to have gone missing]

> -----Original Message-----
> From: spfbis-bounces@ietf.org [mailto:spfbis-bounces@ietf.org] On 
> Behalf Of Dotzero
> Sent: Monday, April 23, 2012 8:14 AM
> To: Murray S. Kucherawy
> Cc: spfbis@ietf.org
> Subject: Re: [spfbis] Review of draft-ietf-spfbis-experiment-05
> 
> You stated that their accuracies are comparable. Given the known 
> weakness of PRA (based on emperical data/testing), that is an 
> incorrect statement.

Suppose PRA was a random number generator rather than a heuristic.  If the data show that SPF and Sender ID thus concur 95% of the time, then I'd say SPF is as accurate as a random number generator.

The result is not an analysis of the compared mechanisms, and we specifically say that already.  It only observes that they concur at a specific rate.  And given that, it doesn't make sense to claim one is substantially more accurate than the other, because it plainly isn't.

> The whole point of SPF (and presumably SIDF) is to mitigate abuse. You 
> have not provided complete details about the dataset you are referring 
> to so it is hard to draw detailed conlusions regarding key
> points:
> 
> 1) What percentage of the dataset does not have a Sender field?
> 2) For data points where there is a Sender field, what percentage of 
> those data points have a Sender field that is aligned with the Mail 
> From (and conceivably From)?
> 3) What percentage of the messages were "abusive"? Would the mail 
> stream selected be a likely target for the particular type of abuse I 
> am pointing out? If not, why would you expect to see this particular 
> type of abuse?

I don't agree that any of these are material to answering the question.

Suppose you're comparing two commercial spam filters.  You don't get to know the guts of them because that's proprietary, but you can observe the black box output, which is to say they each tag x% of mail as spam or not spam.  Now suppose that after your evaluations, the report says both of them tag messages the same way 95% of the time.  I claim that, processing speeds not withstanding, they're basically the same in terms of accuracy.  I'm not saying they are 95% accurate, but I am saying that 95% of the time their accuracy (whatever it is) is the same.  Now, assuming you don't actually care about the difference between 95% and 95.1%, are you going to claim this is an invalid test, merely because you don't know the internal mechanics of the two filters?

> 4) To what extent is the dataset representative of the mail streams 
> that various types and sizes of receivers might receive? That is, is 
> the dataset truly representative or are there potential issues with 
> self selection, etc?

If self-selection of the data renders a data set invalid, then the entire document is invalid.  The data we have come from a handful volunteers who did the work to construct and execute the surveys and report findings.  We have the fortune that at least two of them are sizable participants (Hotmail and Cisco).

If you would like to run such a survey yourself and ensure your view of the world is also represented, or procure such surveys from whatever set of sources you think would make the results more even, then by all means please do so.

-MSK