Outage analysis and report

Glen <glen@amsl.com> Tue, 28 January 2020 23:56 UTC

MIME-Version: 1.0
From: Glen <glen@amsl.com>
Date: Tue, 28 Jan 2020 15:56:40 -0800
Message-ID: <CABL0ig5G0K+ULxAHjLcXw6LicBHutdeOckJ==QMy6=kLZOOhoA@mail.gmail.com>
Subject: Outage analysis and report
To: ietf-announce@ietf.org
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf-announce/Zo3YbZJVpM74fioJ1P6fVKs2NTc>
Precedence: list

Dear IETF Community -

As you know, about 30 hours ago, we moved the IETF to a new server,
containing upgraded OS, software, and a new Python version.  As a part
of that process, the Tools Team moved the Datatracker and their other
software to Python 3.

14 hours ago, that new server suffered a significant data loss. Henrik
was online shortly after the data loss occurred, and called me
immediately. Investigation determined that the loss was caused by a
command in the daily Datatracker cron script.  One rsync command in
that script is designed to make iana yang- parameters available to the
Datatracker.  After the upgrade to Python 3, that script generated a
bad command-line argument, resulting in the rsync command running with
an incorrect target and incorrectly deleting server data.  The missing
data then caused the Datatracker, the Mail Archive, and other tools to
malfunction or fail.  We of course operate a number of hot backup
servers, all of which dutifully picked up the data changes
immediately.

Fortunately, just prior to that script's execution, one of AMS'
offsite backup systems had grabbed a complete copy of the data on the
new server.  So, with the exception of approximately 2 hours of
traffic during which the operating servers were impaired (roughly
0815-1015 GMT Tuesday morning) , no other data was lost. However,
given the estimated time to restore that data (3-4 hours over the
Internet), and given that there could be other unknowns in the
software we hadn't yet identified, the optimal course was clearly to
bring the old server group back online, which I did, restoring service
using the old server approximately 3 hours after the problem started.

At this time, AMS programmers are working on merging yesterday's Mail
archive data into the live archive, while the Tools Team members are
working on merging drafts and Datatracker data and other information
they manage into the live system. We will send an update when this
process is completed, and we will then consult with IETF leadership
and schedule a new cutover event in the near future.

Thank you for your patience.

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)

Outage analysis and report Glen