IETF Server Upgrade/Cutover

Glen <> Fri, 21 February 2020 04:39 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 22B8A120289 for <>; Thu, 20 Feb 2020 20:39:32 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -104.199
X-Spam-Status: No, score=-104.199 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_WHITELIST=-100] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ESgj-42qpQzJ for <>; Thu, 20 Feb 2020 20:39:30 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id F3F4E12013D for <>; Thu, 20 Feb 2020 20:39:29 -0800 (PST)
Received: from (localhost []) by (Postfix) with ESMTPS id 6C846202181 for <>; Thu, 20 Feb 2020 20:38:51 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTPSA id 5114B20217F for <>; Thu, 20 Feb 2020 20:38:51 -0800 (PST)
Received: by with SMTP id h9so883282otj.11 for <>; Thu, 20 Feb 2020 20:39:29 -0800 (PST)
X-Gm-Message-State: APjAAAWReEJ3IBh4Fm7plpVRh19+Kv8YOXQout+BUrrDNfo6k6YS3hsM BV+XRxy9iWRa+/qnn9nTlPbpnY5zupS8JCiA0Qg=
X-Google-Smtp-Source: APXvYqyxIJYmV4acRwD+he6sYQXB4PeEjewewbRY08sTc6t+5cahTn1WW24l97dp+yBK9Ltusi53hhDkm2BHQwOwgPk=
X-Received: by 2002:a9d:7559:: with SMTP id b25mr25806598otl.189.1582259969142; Thu, 20 Feb 2020 20:39:29 -0800 (PST)
MIME-Version: 1.0
References: <>
In-Reply-To: <>
From: Glen <>
Date: Thu, 20 Feb 2020 20:39:16 -0800
X-Gmail-Original-Message-ID: <>
Message-ID: <>
Subject: IETF Server Upgrade/Cutover
Content-Type: text/plain; charset="UTF-8"
Archived-At: <>
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "IETF announcement list. No discussions." <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 21 Feb 2020 04:39:32 -0000

Dear IETF Community:

On Saturday, February 22, we will be re-executing the cutover of all
IETF services to our new virtual production server group. These new
servers contain the latest version of the Linux operating system on
which they run, a new Linux kernel, new versions of many of the open
source software packages we've come to know and love, and Python 3.6,
opening the door for the Tools Team to work additional magic with the
software packages they continue developing for the community.  Last
month, when we performed this process the first time, the process took
about one hour, and to my delight, almost everything worked without
any issues at all.  We did have some minor issues with the
registration system, now corrected, and we had an errant Python script
that ran overnight causing some data loss, which has also been
corrected.  So I'm actually more confident this time, having seen than
pretty much all of our other services survived the upgrade without a
hitch.  Nonetheless, this operation is significant, and will involve
the shutdown of services, a transfer of data, and the subsequent
activation of the new servers.

While the process itself will begin earlier, there will be a visible
one-hour outage window beginning at 9AM Pacific Time (Noon Eastern,
5PM GMT, overnight elsewhere), Saturday February 22.  During this
window, some services will be interrupted at different intervals.
Email and list transport will stop before 9AM, and remain down for
most of the window, while web and Datatracker will be down for shorter
intervals during the window.  The presence or absence of any one
service should not imply expectations for other services during this
interval.  Once we're finished, we will perform a complete reboot,
after which the new servers will be brought online.  As a part of our
testing, we'll send an announcement out again when the work is

Although we've all done quite a bit of work on this (and been through
the process once already) there may still be problems we have not yet
been able to identify.  Both AMS and the Tools Team will be performing
additional checks and testing post-cutover, and we will work to
correct any identified issues as quickly as possible.  We ask for your
patience with this process.  As always, if you have questions or
concerns, you can direct them to the IETF-ACTION desk ( ) which I supervise, or you can contact me
directly via email or Jabber at: , or Telegram or Wire
at @glenams .

Thank you for your patience as we work together to continue to improve
the IETF's online presence, infrastructure and services.

Glen Barney
IT Director
AMS (IETF Secretariat)