Re: [Asrg] Adding a spam button to MUAs

Matthias Leisi <matthias@leisi.net> Mon, 21 December 2009 18:24 UTC

Return-Path: <matthias@leisi.net>
X-Original-To: asrg@core3.amsl.com
Delivered-To: asrg@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3A0B83A6889 for <asrg@core3.amsl.com>; Mon, 21 Dec 2009 10:24:26 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.157
X-Spam-Level:
X-Spam-Status: No, score=0.157 tagged_above=-999 required=5 tests=[BAYES_50=0.001, SUBJECT_FUZZY_TION=0.156]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Nw-S9+ZcHf5G for <asrg@core3.amsl.com>; Mon, 21 Dec 2009 10:24:25 -0800 (PST)
Received: from mail-fx0-f225.google.com (mail-fx0-f225.google.com [209.85.220.225]) by core3.amsl.com (Postfix) with ESMTP id 343103A6405 for <asrg@irtf.org>; Mon, 21 Dec 2009 10:24:25 -0800 (PST)
Received: by fxm25 with SMTP id 25so295755fxm.1 for <asrg@irtf.org>; Mon, 21 Dec 2009 10:24:08 -0800 (PST)
Received: by 10.223.68.155 with SMTP id v27mr10000562fai.10.1261419848367; Mon, 21 Dec 2009 10:24:08 -0800 (PST)
Received: from verleihnix.local (marvin.net.astrum.ch [213.144.132.250]) by mx.google.com with ESMTPS id 22sm8872906fkq.24.2009.12.21.10.24.06 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 21 Dec 2009 10:24:06 -0800 (PST)
Message-ID: <4B2FBD46.1010809@leisi.net>
Date: Mon, 21 Dec 2009 19:24:06 +0100
From: Matthias Leisi <matthias@leisi.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; de; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0
MIME-Version: 1.0
To: asrg@irtf.org
References: <alpine.BSF.2.00.0912082138050.20682@simone.lan> <20091216014800.GA29103@gsp.org> <DBF77720-200E-4846-949F-924388F9CC15@blighty.com> <20091216120742.GA28622@gsp.org> <20091216185904.3B9032421D@panix5.panix.com> <4B296458.5070603@mail-abuse.org> <16C1C8A4-D223-435B-93BC-A9D44F5965A1@guppylake.com> <B14EC7430355853625D0D4EA@lewes.staff.uscs.susx.ac.uk> <BBF2AC03-3C88-4557-9346-343347C196A9@guppylake.com> <240DB04672256506ED548857@lewes.staff.uscs.susx.ac.uk> <4B2A7E8D.8060104@nd.edu> <AF09C4BE1E9DB501F0D2CDF3@paine.local> <940DF2A7-C912-49C1-B967-E82CA69649D3@guppylake.com>
In-Reply-To: <940DF2A7-C912-49C1-B967-E82CA69649D3@guppylake.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [Asrg] Adding a spam button to MUAs
X-BeenThere: asrg@irtf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: Anti-Spam Research Group - IRTF <asrg@irtf.org>
List-Id: Anti-Spam Research Group - IRTF <asrg.irtf.org>
List-Unsubscribe: <http://www.irtf.org/mailman/listinfo/asrg>, <mailto:asrg-request@irtf.org?subject=unsubscribe>
List-Archive: <http://www.irtf.org/mail-archive/web/asrg>
List-Post: <mailto:asrg@irtf.org>
List-Help: <mailto:asrg-request@irtf.org?subject=help>
List-Subscribe: <http://www.irtf.org/mailman/listinfo/asrg>, <mailto:asrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 21 Dec 2009 18:24:26 -0000

Am 21.12.09 18:46, schrieb Nathaniel Borenstein:

> distinction, it's whether the users can.  If the users can't use the
> client's two buttons with sufficiently low error rates, then the
> resulting data can't possibly be useful to the admins.  In other

When I was responsible for spamfilter operation at a former job, error
rates of "human spamfilters" were considerably higher than FP rates of
any possible solution. This is not scientific evidence, but it
illustrates it nicely:

It was very important for the CEO of the company not to lose mail. It
was thus decided that his assistant would go through his inbox and spam
folder and use "Mark as Spam" and "Mark as Not Spam" buttons to clean up
things.

This assistant was very diligent, and highly capable at her job.
Nevertheless, she had a surprisingly high error rate -- for every
hundred mails which she marked as (not) spam, she mis-categorized maybe
five to ten mails.

I did not watch the other users as closely, but they had similar error
rates. The feedback loop from the users could never have been used for
automated actions due to the low quality (maybe it would have been
possible with a larger user base and appropriate statistical analysis).
The buttons were still an important source for fine tuning of the filter
and to find emerging trends in spammer behaviour. Nothing more, but
nothing less.

-- Matthias