[TOOLS-DEVELOPMENT] Mailman subscribe attacks redux

Glen <glen@amsl.com> Tue, 15 September 2015 15:39 UTC

Return-Path: <glen@amsl.com>
X-Original-To: tools-development@ietfa.amsl.com
Delivered-To: tools-development@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7655B1A1A31 for <tools-development@ietfa.amsl.com>; Tue, 15 Sep 2015 08:39:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -103.587
X-Spam-Level:
X-Spam-Status: No, score=-103.587 tagged_above=-999 required=5 tests=[AC_DIV_BONANZA=0.001, BAYES_00=-1.9, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=unavailable
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mnzCOUQkzhFV for <tools-development@ietfa.amsl.com>; Tue, 15 Sep 2015 08:39:24 -0700 (PDT)
Received: from mail.amsl.com (mail.amsl.com [4.31.198.40]) by ietfa.amsl.com (Postfix) with ESMTP id C6F481A1A2F for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:39:20 -0700 (PDT)
Received: from mail.amsl.com (localhost [127.0.0.1]) by c8a.amsl.com (Postfix) with ESMTP id D314A1E5A2A for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:38:44 -0700 (PDT)
Received: from mail-ob0-f172.google.com (mail-ob0-f172.google.com [209.85.214.172]) by c8a.amsl.com (Postfix) with ESMTPSA id AFC2D1E5A2F for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:38:44 -0700 (PDT)
Received: by obbzf10 with SMTP id zf10so82604515obb.2 for <tools-development@ietf.org>; Tue, 15 Sep 2015 08:39:20 -0700 (PDT)
X-Received: by 10.60.125.8 with SMTP id mm8mr19331912oeb.73.1442331560015; Tue, 15 Sep 2015 08:39:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.202.102.71 with HTTP; Tue, 15 Sep 2015 08:39:00 -0700 (PDT)
From: Glen <glen@amsl.com>
Date: Tue, 15 Sep 2015 08:39:00 -0700
Message-ID: <CABL0ig6eV=JhUyLMAWDtJfYmTAPB8_oax8MgNZsgcv5mBJdcTg@mail.gmail.com>
To: Glen Barney <glen@amsl.com>
Content-Type: multipart/alternative; boundary="047d7b339b21487fbb051fcafc42"
Archived-At: <http://mailarchive.ietf.org/arch/msg/tools-development/p4iI5kxO-J1LZR5CQLerZYflHc0>
X-Mailman-Approved-At: Tue, 15 Sep 2015 08:40:37 -0700
Subject: [TOOLS-DEVELOPMENT] Mailman subscribe attacks redux
X-BeenThere: tools-development@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: glen@amsl.com
List-Id: Tools Development list server <tools-development.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tools-development>, <mailto:tools-development-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tools-development/>
List-Post: <mailto:tools-development@ietf.org>
List-Help: <mailto:tools-development-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tools-development>, <mailto:tools-development-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 15 Sep 2015 15:39:25 -0000

Good day:

You may recall that we have been experiencing an attack against Mailman, in
which a large number of automated subscribe requests to addresses of the
form "someroot+somesuffix@somedomain.org" are being sent by compromised
computers worldwide to the IETF's Mailman server.  A form of
Denial-of-Service, this attack has affected our lists and list admins, and
has caused us to generate a ton of outgoing/response mail that should not
be sent in the first place.

The volume of the attack has previously been low and manageable.  However,
overnight, the level of subscribe attacks against the IETF Mailman server
increased dramatically, to the point that Mailman's bounce processors were
getting overwhelmed and dying, triggering alarms and compromising mail
operations.

To answer some obvious questions:  This attack is distributed. Over 4000 IP
addresses from just last night were participating.  The addresses cannot be
blocked by normal automated means on our server, because we use
Cloudflare.  We would have to feed the addresses into Cloudflare, and that
is a laborious, manual process that does not scale.  And as we learned from
previous interactions with Cloudflare, there is not much that can be done
about attacks of this type from their side.

When this began, we provided expressions and guidance that would allow list
owners to ban these addresses on a per-list basis; alas, the adoption rate
of those recommendations was too low, and the attack increased too quickly.

Because a service outage is imminent, I have now instructed the servers to
reject all subscribe requests from all email addresses containing a "+"
sign.  Users posting subscribe requests containing a plus sign are referred
to the secretariat to complete their request.  The request is halted after
that message is displayed, and no further processing, notification of list
owners, or outgoing mail is generated: the request is deleted as an error.

Users already subscribed to lists using valid addresses of this form are
not affected.  They can still manage and remove their subscriptions
directly.  This only affects new requests... our server will no longer
attempt to process new requests containing + signs.  This behavior is not
optional or per-list, it applies server-wide and cannot be overridden on a
per-list basis.

While I note that some users (roughly 1011 "+" addresses currently exist in
Mailman, out of the 77809 total subscribers present at the moment) use this
address form for filtering, I note that Mailman also provides list-based
RFC2369/2919 headers that allow for filtering as well.  And I wish to make
clear again that we are not *refusing* subscriptions of this type - we are
just referring them for manual processing.

These changes are deployed as of now, and are effective for all IETF
mailing lists, regardless of list configuration settings.

I apologize for the inconvenience this will cause to new users wanting this
address format; however, the volume of the attack has left me with no other
choice.

As always, any questions, please let me know.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)