Re: [Asrg] Adding a spam button to MUAs

Jose-Marcio Martins da Cruz <Jose-Marcio.Martins@mines-paristech.fr> Mon, 21 December 2009 14:33 UTC

Return-Path: <Jose-Marcio.Martins@mines-paristech.fr>
X-Original-To: asrg@core3.amsl.com
Delivered-To: asrg@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id DB3F83A6767 for <asrg@core3.amsl.com>; Mon, 21 Dec 2009 06:33:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.414
X-Spam-Level:
X-Spam-Status: No, score=0.414 tagged_above=-999 required=5 tests=[AWL=0.093, BAYES_40=-0.185, HELO_EQ_FR=0.35, SUBJECT_FUZZY_TION=0.156]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qkU2KlMk0lz8 for <asrg@core3.amsl.com>; Mon, 21 Dec 2009 06:33:12 -0800 (PST)
Received: from boipeva.ensmp.fr (cobra.ensmp.fr [194.214.158.101]) by core3.amsl.com (Postfix) with ESMTP id D299F3A67D9 for <asrg@irtf.org>; Mon, 21 Dec 2009 06:33:11 -0800 (PST)
Received: from localhost.localdomain (joe.j-chkmail.org [88.168.143.55]) (authenticated bits=0) by boipeva.ensmp.fr (8.14.3/8.14.3/JMMC-11/Feb/2009) with ESMTP id nBLEWpXL007707 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <asrg@irtf.org>; Mon, 21 Dec 2009 15:32:51 +0100 (MET)
Message-ID: <4B2F8709.4020903@mines-paristech.fr>
Date: Mon, 21 Dec 2009 15:32:41 +0100
From: Jose-Marcio Martins da Cruz <Jose-Marcio.Martins@mines-paristech.fr>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090908 Fedora/1.1.18-1.fc11 SeaMonkey/1.1.18
MIME-Version: 1.0
To: Anti-Spam Research Group - IRTF <asrg@irtf.org>
References: <alpine.BSF.2.00.0912082138050.20682@simone.lan> <20091216014800.GA29103@gsp.org> <DBF77720-200E-4846-949F-924388F9CC15@blighty.com> <20091216120742.GA28622@gsp.org> <20091216185904.3B9032421D@panix5.panix.com> <4B296458.5070603@mail-abuse.org> <16C1C8A4-D223-435B-93BC-A9D44F5965A1@guppylake.com> <B14EC7430355853625D0D4EA@lewes.staff.uscs.susx.ac.uk> <BBF2AC03-3C88-4557-9346-343347C196A9@guppylake.com> <240DB04672256506ED548857@lewes.staff.uscs.susx.ac.uk> <4B2A7E8D.8060104@nd.edu> <20091217200605.8E99E2421D@panix5.panix.com> <4B2B0E4B.3050509@dcrocker.net> <4B2E1E76.9000400@mines-paristech.fr> <A8792A43052B049E5A4851E5@lewes.staff.uscs.susx.ac.uk>
In-Reply-To: <A8792A43052B049E5A4851E5@lewes.staff.uscs.susx.ac.uk>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Miltered: at boipeva with ID 4B2F8713.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)!
X-j-chkmail-Enveloppe: 4B2F8713.000/88.168.143.55/joe.j-chkmail.org/localhost.localdomain/<Jose-Marcio.Martins@mines-paristech.fr>
Subject: Re: [Asrg] Adding a spam button to MUAs
X-BeenThere: asrg@irtf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: Jose-Marcio.Martins@mines-paristech.fr, Anti-Spam Research Group - IRTF <asrg@irtf.org>
List-Id: Anti-Spam Research Group - IRTF <asrg.irtf.org>
List-Unsubscribe: <http://www.irtf.org/mailman/listinfo/asrg>, <mailto:asrg-request@irtf.org?subject=unsubscribe>
List-Archive: <http://www.irtf.org/mail-archive/web/asrg>
List-Post: <mailto:asrg@irtf.org>
List-Help: <mailto:asrg-request@irtf.org?subject=help>
List-Subscribe: <http://www.irtf.org/mailman/listinfo/asrg>, <mailto:asrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 21 Dec 2009 14:33:13 -0000

Ian Eiloart wrote:

...

>>
>> These error rates are most of the time bigger than what can be achieved
>> by spam filters. So it's probably a bad idea to consider that user
>> feedback is reliable. User interface shall be as simple as possible.
> 
> But wait, we're talking about messages that the spam filter hasn't 
> rejected. Additional data *has* to be useful.

Sure, I've just mentionned that user feedback isn't reliable. There are many reasons. One thing to
do to mitigate user feedback lack of reliability is to make user interface as "good" as possible.

> 
> The false positive rates are only a problem if the admin stupid enough 
> to consider a single report as definitive. If you deliver a message to 
> 100 users, and three report it as spam, then you probably take no 
> action. If 20 report it as spam, then you need to take a closer look.

Agree, but if 20 report it as spam, but if the other 80 reported it as ham or if they didn't
reported anything these are different situations.

> I certainly don't think a 7% error rate is enough to determine that 
> users should not be given the opportunity to distinguish between 
> unwanted mail and reportable junk.

Sure but... I don't think about a binary decision : using or not user feedback, nor trying to 
correct user feedback till have a perfect clean feedback. The idea behind Cormack paper is to handle 
user feedback the same was as a noisy signal. Try to correct it trivial errors, but accept existence 
of noise and make your system as insensitive as possible to noise.

I made an experiment with some users recently. In this particular experiment, I noticed some obvious 
things : people are concerned about wanted and unwanted mail - ham and spam isn't their problem; 
appreciation for wanted/unwanted messages varies not only from user to user but also depends on the 
moment. The latter means that, at different times, the same messages won't be judged the same way by 
the same user. Also, error rates aren't the same for different kind of messages, even inside the 
same class.

I don't generalize this and, unfortunately, I don't have numbers reliable enough to publish, but I'd 
like to have. I don't believe in opinions which aren't supported by reliable numbers.

> You can also combine reporting rates with your bayesian content analyser 
> or spamassassin score, or with your reputational score for the sender 
> domain, etc.

Sure, but even after that, you still won't have a perfectly clean user feedback.


--