[93attendees] Major Outage Update

Jim Martin <jrmii@isc.org> Thu, 23 July 2015 16:20 UTC

Return-Path: <jrmii@isc.org>
X-Original-To: 93attendees@ietfa.amsl.com
Delivered-To: 93attendees@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 986681A01F2 for <93attendees@ietfa.amsl.com>; Thu, 23 Jul 2015 09:20:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.911
X-Spam-Level:
X-Spam-Status: No, score=-6.911 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6rpwCrngBvnW for <93attendees@ietfa.amsl.com>; Thu, 23 Jul 2015 09:20:22 -0700 (PDT)
Received: from mx.ams1.isc.org (mx.ams1.isc.org [199.6.1.65]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E1E2A1A1A42 for <93attendees@ietf.org>; Thu, 23 Jul 2015 09:20:21 -0700 (PDT)
Received: from zmx1.isc.org (zmx1.isc.org [149.20.0.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx.ams1.isc.org (Postfix) with ESMTPS id 9FDA41FCAD3 for <93attendees@ietf.org>; Thu, 23 Jul 2015 16:20:15 +0000 (UTC)
Received: from zmx1.isc.org (localhost [127.0.0.1]) by zmx1.isc.org (Postfix) with ESMTPS id 06F3F16008E for <93attendees@ietf.org>; Thu, 23 Jul 2015 16:21:12 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1]) by zmx1.isc.org (Postfix) with ESMTP id E257616008D for <93attendees@ietf.org>; Thu, 23 Jul 2015 16:21:11 +0000 (UTC)
Received: from zmx1.isc.org ([127.0.0.1]) by localhost (zmx1.isc.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id exreFuDb2kzb for <93attendees@ietf.org>; Thu, 23 Jul 2015 16:21:11 +0000 (UTC)
Received: from dhcp-89de.meeting.ietf.org (unknown [31.133.138.222]) by zmx1.isc.org (Postfix) with ESMTPSA id 6B6C0160044 for <93attendees@ietf.org>; Thu, 23 Jul 2015 16:21:11 +0000 (UTC)
From: Jim Martin <jrmii@isc.org>
X-Pgp-Agent: GPGMail 2.5
Content-Type: multipart/signed; boundary="Apple-Mail=_02D8ADDA-139D-4225-B171-4E02B58037FF"; protocol="application/pgp-signature"; micalg=pgp-sha256
Date: Thu, 23 Jul 2015 18:20:11 +0200
Message-Id: <A2EFA276-4257-40F4-ACA4-FDF8815FAEA6@isc.org>
To: 93attendees@ietf.org
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
X-Mailer: Apple Mail (2.2102)
Archived-At: <http://mailarchive.ietf.org/arch/msg/93attendees/emC_w8tUUj5THGFu9ItcOG64N_U>
Subject: [93attendees] Major Outage Update
X-BeenThere: 93attendees@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Mailing list of IETF 93 attendees that have opted in on this list. " <93attendees.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/93attendees>, <mailto:93attendees-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/93attendees/>
List-Post: <mailto:93attendees@ietf.org>
List-Help: <mailto:93attendees-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/93attendees>, <mailto:93attendees-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 23 Jul 2015 16:20:23 -0000

Gentlepeople,
	We believe that most, if not all, services have been restored. We had a catastrophic failure of half of the core switch stack, and the remaining half became very confused. After a power cycle, we migrated all of the links to the edge switches from the dead switch to the good one, and adjusted the configs to account for the changes. The upshot is that it took a while, but we think we’re back. Please give us another 30 minutes to clean things up, but after that, let us know if you see issues.

	- Jim