New server failure

[Glen <[e-mail]>]

All -

It is with deep regret that I announce that we encountered a system
failure on the new server overnight.

Three hours ago, at 00:15 Pacific time, several key data directories
on the new server were wiped out, almost as if a synchronization job
had failed or was overwriting data.  The damage was not in an
identifiable pattern.  An initial investigation was performed without
an immediate result.  Missing directories seemed to be random.

After consulting with Henrik, we felt we had no choice but to bring
the old server back online, and we have done so.

Although this doesn't appear to be a hacker/attack, I have not
dismissed that possibility, and have taken appropriate precautions on
the old server in the event foul play was involved.  Instead I suspect
that something pertaining to our software upgrade work was the culprit
here, but it may take a while before we identify the cause.

What this means, for now, unfortunately, is that actions taken
yesterday, within the past 14 hours from the time of this message,
including emails sent and files uploaded, are lost.  If you uploaded a
draft yesterday, or took any other Datatracker action, you will need
to repeat those actions today.

Unfortunately whatever occurred was immediately propagated to the
backup machines, since the servers didn't crash.  But in addition to
those things, I have a secondary backup system that runs daily.  We
have files and email archive messages up to about 4 hours ago, and
we'll be working to restore those as soon as we can.

In the meantime, the old servers are back up and running, and we will
investigate and determine a way forward as soon as possible.

Thank you for your patience.

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)