[109all] Initial report on yesterday's jabber service interruption

IETF Executive Director <exec-director@ietf.org> Fri, 20 November 2020 00:43 UTC

Return-Path: <exec-director@ietf.org>
X-Original-To: 109all@ietfa.amsl.com
Delivered-To: 109all@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 2A6973A1433 for <109all@ietfa.amsl.com>; Thu, 19 Nov 2020 16:43:24 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id Xp2LuViNb666 for <109all@ietfa.amsl.com>; Thu, 19 Nov 2020 16:43:23 -0800 (PST)
Received: from jays-mbp.localdomain (unknown []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPSA id EFB253A142D for <109all@ietf.org>; Thu, 19 Nov 2020 16:43:22 -0800 (PST)
From: IETF Executive Director <exec-director@ietf.org>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.\))
Message-Id: <147DC95E-04AF-4258-B1F4-0433844A1498@ietf.org>
Date: Fri, 20 Nov 2020 13:43:20 +1300
To: 109all@ietf.org
X-Mailer: Apple Mail (2.3608.
Archived-At: <https://mailarchive.ietf.org/arch/msg/109all/3Z-iW6LBKQbcqVaKRMGuKmZTwdk>
Subject: [109all] Initial report on yesterday's jabber service interruption
X-BeenThere: 109all@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Official communication about IETF 109 <109all.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/109all>, <mailto:109all-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/109all/>
List-Post: <mailto:109all@ietf.org>
List-Help: <mailto:109all-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/109all>, <mailto:109all-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Nov 2020 00:43:24 -0000

As previously reported, there was an interruption to the jabber service during yesterday’s break between sessions 2 and 3.  This has now been diagnosed as a DoS condition, quite possibly unintentional, on our jabber server.  

The IETF jabber server is configured to accept 32,000 connections in total and a maximum of 100 messages/sec.  This is an order of magnitude higher than our expected levels during an IETF meeting.  Yesterday the server received a spike in connections at 17,000/sec and a spike in messages at 4000/sec that continued after the limits had been reached and until the server died a few minutes later.  When the server restarted the spike was no longer present.

Work is ongoing to identify the source(s) of the spike and to detect and mitigate any further spikes.


Jay Daley
IETF Executive Director