[Pce] Fwd: New server failure
<julien.meuric@orange.com> Tue, 28 January 2020 13:12 UTC
Return-Path: <julien.meuric@orange.com>
X-Original-To: pce@ietfa.amsl.com
Delivered-To: pce@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 721F012006B for <pce@ietfa.amsl.com>; Tue, 28 Jan 2020 05:12:38 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zOe2jYjgv_6D for <pce@ietfa.amsl.com>; Tue, 28 Jan 2020 05:12:36 -0800 (PST)
Received: from relais-inet.orange.com (relais-inet.orange.com [80.12.66.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3CD05120058 for <pce@ietf.org>; Tue, 28 Jan 2020 05:12:36 -0800 (PST)
Received: from opfedar00.francetelecom.fr (unknown [xx.xx.xx.11]) by opfedar20.francetelecom.fr (ESMTP service) with ESMTP id 486RqQ6pthz8tFH for <pce@ietf.org>; Tue, 28 Jan 2020 14:12:34 +0100 (CET)
Received: from Exchangemail-eme6.itn.ftgroup (unknown [xx.xx.13.104]) by opfedar00.francetelecom.fr (ESMTP service) with ESMTP id 486RqQ5xB3zCql7 for <pce@ietf.org>; Tue, 28 Jan 2020 14:12:34 +0100 (CET)
Received: from [10.193.71.21] (10.114.13.247) by OPEXCAUBM5F.corporate.adroot.infra.ftgroup (10.114.13.104) with Microsoft SMTP Server (TLS) id 14.3.468.0; Tue, 28 Jan 2020 14:12:34 +0100
References: <CABL0ig6Exs_fTKNqDNP2UwJarhzUCq=AbDOB_pwy6YB3_AMK+Q@mail.gmail.com>
To: "pce@ietf.org" <pce@ietf.org>
From: julien.meuric@orange.com
Organization: Orange
X-Forwarded-Message-Id: <CABL0ig6Exs_fTKNqDNP2UwJarhzUCq=AbDOB_pwy6YB3_AMK+Q@mail.gmail.com>
Message-ID: <bde03df4-b48b-db49-48ec-5e3a88eacdfd@orange.com>
Date: Tue, 28 Jan 2020 14:12:30 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0
MIME-Version: 1.0
In-Reply-To: <CABL0ig6Exs_fTKNqDNP2UwJarhzUCq=AbDOB_pwy6YB3_AMK+Q@mail.gmail.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg="sha-256"; boundary="------------ms000403010208010405050609"
X-Originating-IP: [10.114.13.247]
Archived-At: <https://mailarchive.ietf.org/arch/msg/pce/RDa5-nPkqdmVZBSKq4iSB1opyRA>
Subject: [Pce] Fwd: New server failure
X-BeenThere: pce@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Path Computation Element <pce.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/pce>, <mailto:pce-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/pce/>
List-Post: <mailto:pce@ietf.org>
List-Help: <mailto:pce-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/pce>, <mailto:pce-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 28 Jan 2020 13:12:39 -0000
FYI -------- Forwarded Message -------- Date: Tue, 28 Jan 2020 03:38:01 -0800 From: Glen <glen@amsl.com> All - It is with deep regret that I announce that we encountered a system failure on the new server overnight. Three hours ago, at 00:15 Pacific time, several key data directories on the new server were wiped out, almost as if a synchronization job had failed or was overwriting data. The damage was not in an identifiable pattern. An initial investigation was performed without an immediate result. Missing directories seemed to be random. After consulting with Henrik, we felt we had no choice but to bring the old server back online, and we have done so. Although this doesn't appear to be a hacker/attack, I have not dismissed that possibility, and have taken appropriate precautions on the old server in the event foul play was involved. Instead I suspect that something pertaining to our software upgrade work was the culprit here, but it may take a while before we identify the cause. What this means, for now, unfortunately, is that actions taken yesterday, within the past 14 hours from the time of this message, including emails sent and files uploaded, are lost. If you uploaded a draft yesterday, or took any other Datatracker action, you will need to repeat those actions today. Unfortunately whatever occurred was immediately propagated to the backup machines, since the servers didn't crash. But in addition to those things, I have a secondary backup system that runs daily. We have files and email archive messages up to about 4 hours ago, and we'll be working to restore those as soon as we can. In the meantime, the old servers are back up and running, and we will investigate and determine a way forward as soon as possible. Thank you for your patience. Glen -- Glen Barney IT Director AMS (IETF Secretariat) _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www.ietf.org/mailman/listinfo/ietf-announce
- [Pce] Fwd: New server failure julien.meuric