Re: IETF Service Outage

Glen <glen@amsl.com> Wed, 23 March 2016 13:27 UTC

MIME-Version: 1.0
In-Reply-To: <56F29132.4010109@cisco.com>
References: <CABL0ig4s9PBVS5icz4y=M2nJXtcPH9iVFEptooLsRUCnUjprMg@mail.gmail.com> <56F29132.4010109@cisco.com>
From: Glen <glen@amsl.com>
Date: Wed, 23 Mar 2016 06:13:12 -0700
Message-ID: <CABL0ig468ZDB3tAA0C0NPyP553CH8Knq4bzobD_Pt2p3rE35uQ@mail.gmail.com>
Subject: Re: IETF Service Outage
To: Eliot Lear <lear@cisco.com>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <http://mailarchive.ietf.org/arch/msg/ietf/gQmQvYh8_Atwz_2NOP1m6MJRSVE>
Cc: ietf <ietf@ietf.org>
Precedence: list
Reply-To: glen@amsl.com

On Wed, Mar 23, 2016 at 5:50 AM, Eliot Lear <lear@cisco.com> wrote:
> I'm curious- do you have sufficient data to perform an analysis to
> determine the source of the service degradation?

Hi Eliot -

We're still looking, but, at the moment, no, it does not appear that way.

Our servers run Linux, our switches are Cisco, and log levels on
everything are quite high.  Our logs are full of noise about all kinds
of other, unrelated, server activities; but, as often seems to be the
case, the logs are silent about things related to this issue.  To my
great frustration.

Our engineers are always looking at additional ways of monitoring
things, but whether this was some kind of denial-of-service attack
against a physical host, or an OS failure of some kind, or maybe just
a bad network cable, we can't yet tell.

When things like this happen unexpectedly, we only want to take a
minimal amount of time to try and perform testing mid-event.  Having
done a number of tests, checks, and localized reset procedures, we
were just about to do a reboot of the physical host when the network
just came back to normal, all by itself.  So, this time, so far, I am
unable to determine a source, which, of course, is probably the most
frustrating outcome possible.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)