[TOOLS-DEVELOPMENT] Mailman subscribe attacks redux
Glen <glen@amsl.com> Tue, 15 September 2015 15:39 UTC
Return-Path: <glen@amsl.com>
X-Original-To: tools-development@ietfa.amsl.com
Delivered-To: tools-development@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7655B1A1A31 for <tools-development@ietfa.amsl.com>; Tue, 15 Sep 2015 08:39:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -103.587
X-Spam-Level:
X-Spam-Status: No, score=-103.587 tagged_above=-999 required=5 tests=[AC_DIV_BONANZA=0.001, BAYES_00=-1.9, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=unavailable
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mnzCOUQkzhFV for <tools-development@ietfa.amsl.com>; Tue, 15 Sep 2015 08:39:24 -0700 (PDT)
Received: from mail.amsl.com (mail.amsl.com [4.31.198.40]) by ietfa.amsl.com (Postfix) with ESMTP id C6F481A1A2F for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:39:20 -0700 (PDT)
Received: from mail.amsl.com (localhost [127.0.0.1]) by c8a.amsl.com (Postfix) with ESMTP id D314A1E5A2A for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:38:44 -0700 (PDT)
Received: from mail-ob0-f172.google.com (mail-ob0-f172.google.com [209.85.214.172]) by c8a.amsl.com (Postfix) with ESMTPSA id AFC2D1E5A2F for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:38:44 -0700 (PDT)
Received: by obbzf10 with SMTP id zf10so82604515obb.2 for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:39:20 -0700 (PDT)
X-Received: by 10.60.125.8 with SMTP id mm8mr19331912oeb.73.1442331560015; Tue, 15 Sep 2015 08:39:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.202.102.71 with HTTP; Tue, 15 Sep 2015 08:39:00 -0700 (PDT)
From: Glen <glen@amsl.com>
Date: Tue, 15 Sep 2015 08:39:00 -0700
Message-ID: <CABL0ig6eV=JhUyLMAWDtJfYmTAPB8_oax8MgNZsgcv5mBJdcTg@mail.gmail.com>
To: Glen Barney <glen@amsl.com>
Content-Type: multipart/alternative; boundary="047d7b339b21487fbb051fcafc42"
Archived-At: <http://mailarchive.ietf.org/arch/msg/tools-development/p4iI5kxO-J1LZR5CQLerZYflHc0>
X-Mailman-Approved-At: Tue, 15 Sep 2015 08:40:37 -0700
Subject: [TOOLS-DEVELOPMENT] Mailman subscribe attacks redux
X-BeenThere: tools-development@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: glen@amsl.com
List-Id: Tools Development list server <tools-development.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tools-development>, <mailto:tools-development-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tools-development/>
List-Post: <mailto:tools-development@ietf.org>
List-Help: <mailto:tools-development-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tools-development>, <mailto:tools-development-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 15 Sep 2015 15:39:25 -0000
Good day: You may recall that we have been experiencing an attack against Mailman, in which a large number of automated subscribe requests to addresses of the form "someroot+somesuffix@somedomain.org" are being sent by compromised computers worldwide to the IETF's Mailman server. A form of Denial-of-Service, this attack has affected our lists and list admins, and has caused us to generate a ton of outgoing/response mail that should not be sent in the first place. The volume of the attack has previously been low and manageable. However, overnight, the level of subscribe attacks against the IETF Mailman server increased dramatically, to the point that Mailman's bounce processors were getting overwhelmed and dying, triggering alarms and compromising mail operations. To answer some obvious questions: This attack is distributed. Over 4000 IP addresses from just last night were participating. The addresses cannot be blocked by normal automated means on our server, because we use Cloudflare. We would have to feed the addresses into Cloudflare, and that is a laborious, manual process that does not scale. And as we learned from previous interactions with Cloudflare, there is not much that can be done about attacks of this type from their side. When this began, we provided expressions and guidance that would allow list owners to ban these addresses on a per-list basis; alas, the adoption rate of those recommendations was too low, and the attack increased too quickly. Because a service outage is imminent, I have now instructed the servers to reject all subscribe requests from all email addresses containing a "+" sign. Users posting subscribe requests containing a plus sign are referred to the secretariat to complete their request. The request is halted after that message is displayed, and no further processing, notification of list owners, or outgoing mail is generated: the request is deleted as an error. Users already subscribed to lists using valid addresses of this form are not affected. They can still manage and remove their subscriptions directly. This only affects new requests... our server will no longer attempt to process new requests containing + signs. This behavior is not optional or per-list, it applies server-wide and cannot be overridden on a per-list basis. While I note that some users (roughly 1011 "+" addresses currently exist in Mailman, out of the 77809 total subscribers present at the moment) use this address form for filtering, I note that Mailman also provides list-based RFC2369/2919 headers that allow for filtering as well. And I wish to make clear again that we are not *refusing* subscriptions of this type - we are just referring them for manual processing. These changes are deployed as of now, and are effective for all IETF mailing lists, regardless of list configuration settings. I apologize for the inconvenience this will cause to new users wanting this address format; however, the volume of the attack has left me with no other choice. As always, any questions, please let me know. Glen Glen Barney IT Director AMS (IETF Secretariat)