Final outage update / conclusion / summary

Glen <glen@amsl.com> Wed, 22 April 2020 19:17 UTC

Return-Path: <glen@amsl.com>
X-Original-To: ietf-announce@ietfa.amsl.com
Delivered-To: ietf-announce@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3FEBE3A0B62 for <ietf-announce@ietfa.amsl.com>; Wed, 22 Apr 2020 12:17:32 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -101.9
X-Spam-Level:
X-Spam-Status: No, score=-101.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NIpsl0e2kPIB for <ietf-announce@ietfa.amsl.com>; Wed, 22 Apr 2020 12:17:30 -0700 (PDT)
Received: from mail.amsl.com (c8a.amsl.com [4.31.198.40]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B70B53A0B55 for <ietf-announce@ietf.org>; Wed, 22 Apr 2020 12:17:30 -0700 (PDT)
Received: from mail.amsl.com (localhost [127.0.0.1]) by c8a.amsl.com (Postfix) with ESMTPS id 859FD2044F6 for <ietf-announce@ietf.org>; Wed, 22 Apr 2020 12:16:43 -0700 (PDT)
Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) by c8a.amsl.com (Postfix) with ESMTPSA id 6A29D2044F5 for <ietf-announce@ietf.org>; Wed, 22 Apr 2020 12:16:43 -0700 (PDT)
Received: by mail-ot1-f51.google.com with SMTP id c3so3194088otp.8 for <ietf-announce@ietf.org>; Wed, 22 Apr 2020 12:17:30 -0700 (PDT)
X-Gm-Message-State: AGi0PuZEbJjLBse+o8Ojtb2IioVGKt4QBmLOcDy39my4F2lNA9QE8b5b 77sypqZjepBPNwJKN/gJkptyNI2VK898ErGvJ3c=
X-Google-Smtp-Source: APiQypJu7NS6lW3rarqfnS7qyN2v4PQvO5IEP5AiucEd9ea+AdsZrNlHjAArOHk7xe1qcRyXy6cWh4XqyRW9Rvab/Uc=
X-Received: by 2002:a05:6830:459:: with SMTP id d25mr481079otc.189.1587583049949; Wed, 22 Apr 2020 12:17:29 -0700 (PDT)
MIME-Version: 1.0
From: Glen <glen@amsl.com>
Date: Wed, 22 Apr 2020 12:17:16 -0700
X-Gmail-Original-Message-ID: <CABL0ig6WUA_hRrdw5J-BYugCnPZbDB=XjRafpxH5BOOuWn_j8Q@mail.gmail.com>
Message-ID: <CABL0ig6WUA_hRrdw5J-BYugCnPZbDB=XjRafpxH5BOOuWn_j8Q@mail.gmail.com>
Subject: Final outage update / conclusion / summary
To: ietf-announce@ietf.org
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf-announce/FoBu0GgNGD7G_S5TmJakjpQHyC0>
X-BeenThere: ietf-announce@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "IETF announcement list. No discussions." <ietf-announce.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf-announce>, <mailto:ietf-announce-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf-announce/>
List-Post: <mailto:ietf-announce@ietf.org>
List-Help: <mailto:ietf-announce-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf-announce>, <mailto:ietf-announce-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 22 Apr 2020 19:17:33 -0000

Dear IETF Community -

Overnight, I received the following update, which I'm sharing for
interest only, as a result of many inquiries from the community.  The
outage is now over, and all seems to be well.  I do not expect to
receive or have any further information on this (unless they in the
future perform further repairs and notify me about them, in which case
those will of course be shared as well.)

Thank you all for your support during this incident!

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)

--snip--

IP NOC reports splicing at the second location, Milwaukee, WI on the
96 count cable has been completed. Associated network alarms and
network congestion has cleared. Field Operations advise this was a
temporary restoration to expedite restoral. A permanent repair will be
conducted at a later date during a scheduled maintenance window.
Impacted clients will be notified prior to the work being conducted.
Field Teams at the first location in Fort Worth, TX continue
restoration efforts however no further impact to services is
anticipated.

Event Conclusion Summary

Outage Start: April 21, 2020 at 13:55 GMT
Outage Stop: April 21, 2020 at 20:45 GMT

Root Cause: Multiple fiber cuts were impacting IP services in various markets.

Fix Action: The cut fiber was repaired, thus restoring services.

This service impact has concluded.