New server failure

Glen <> Tue, 28 January 2020 11:38 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 2A5FA12004E for <>; Tue, 28 Jan 2020 03:38:15 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -104.2
X-Spam-Status: No, score=-104.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id uHWsAK-Iyddl for <>; Tue, 28 Jan 2020 03:38:13 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id D2C741200C4 for <>; Tue, 28 Jan 2020 03:38:13 -0800 (PST)
Received: from (localhost []) by (Postfix) with ESMTPS id 0B1C52039B4 for <>; Tue, 28 Jan 2020 03:37:46 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTPSA id E395C2039B3 for <>; Tue, 28 Jan 2020 03:37:45 -0800 (PST)
Received: by with SMTP id r16so11674335otd.2 for <>; Tue, 28 Jan 2020 03:38:13 -0800 (PST)
X-Gm-Message-State: APjAAAXz14PTP32RtjnoR7m6fRbKh7RM/hfoXrqM+M6uBYCDOMvCK7qU zpzhZhYSMrN6pxsmmQ+xdXyW/Eb3upVQkibVhuw=
X-Google-Smtp-Source: APXvYqy3hKD6lLCoRF8UiSxpkYdxrvT50b1JlDVi1LH5C2ijAS1UJbCr2fpZvnf/dWIxCemsP5Cztuo4nXFu/XGIgTs=
X-Received: by 2002:a05:6830:112:: with SMTP id i18mr15648713otp.209.1580211493006; Tue, 28 Jan 2020 03:38:13 -0800 (PST)
MIME-Version: 1.0
From: Glen <>
Date: Tue, 28 Jan 2020 03:38:01 -0800
X-Gmail-Original-Message-ID: <>
Message-ID: <>
Subject: New server failure
Content-Type: text/plain; charset="UTF-8"
Archived-At: <>
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "IETF announcement list. No discussions." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 28 Jan 2020 11:38:19 -0000

All -

It is with deep regret that I announce that we encountered a system
failure on the new server overnight.

Three hours ago, at 00:15 Pacific time, several key data directories
on the new server were wiped out, almost as if a synchronization job
had failed or was overwriting data.  The damage was not in an
identifiable pattern.  An initial investigation was performed without
an immediate result.  Missing directories seemed to be random.

After consulting with Henrik, we felt we had no choice but to bring
the old server back online, and we have done so.

Although this doesn't appear to be a hacker/attack, I have not
dismissed that possibility, and have taken appropriate precautions on
the old server in the event foul play was involved.  Instead I suspect
that something pertaining to our software upgrade work was the culprit
here, but it may take a while before we identify the cause.

What this means, for now, unfortunately, is that actions taken
yesterday, within the past 14 hours from the time of this message,
including emails sent and files uploaded, are lost.  If you uploaded a
draft yesterday, or took any other Datatracker action, you will need
to repeat those actions today.

Unfortunately whatever occurred was immediately propagated to the
backup machines, since the servers didn't crash.  But in addition to
those things, I have a secondary backup system that runs daily.  We
have files and email archive messages up to about 4 hours ago, and
we'll be working to restore those as soon as we can.

In the meantime, the old servers are back up and running, and we will
investigate and determine a way forward as soon as possible.

Thank you for your patience.

Glen Barney
IT Director
AMS (IETF Secretariat)