[109all] NOC update

Sean Croghan <sean@xagsolutions.com> Wed, 18 November 2020 00:56 UTC

Return-Path: <sean@xagsolutions.com>
X-Original-To: 109all@ietfa.amsl.com
Delivered-To: 109all@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D3F5A3A1168 for <109all@ietfa.amsl.com>; Tue, 17 Nov 2020 16:56:55 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D2wrNHaXPbvG for <109all@ietfa.amsl.com>; Tue, 17 Nov 2020 16:56:54 -0800 (PST)
Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2101.outbound.protection.outlook.com [40.107.92.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5406D3A1167 for <109all@ietf.org>; Tue, 17 Nov 2020 16:56:54 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UG+9ljbotjrDTCmZQarpqW7UXY5csuBqGFviFM17FFnAN+Iy+ooIIsxIcojVI9PhXnQMoIZv4LVKE3WXSNhEe1vaAZloWD1sXIQi3r1APmC1PmuKum0U+uyjiHnBhgai1V0c9WWXY5SmESQOOEfji9W9DDFqxz6pm5NzRaCBrpYpMy9sFzMn5ZLiFnH3FyrSqHhBlwstJB2d+9uqxI4ytXls0dCDm7Vu6ulCN4NVG9Yc6oQ23EGuS6ehjJQ1xQAnjVxbPhZmGQpXibYTiq6E/sO9Zbpp8GaVd5n1kvs0bdMlPiRr0b4KOt6+yPKi0qQaoxcxN9rJWGUU7XlLmocuUA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ovVTtVHeXSxuxsQssiT5hsvjuPsNSn3PoPWYwJbKDvk=; b=TlLYVUGbCOf4Icp8Vqjii3b4p/7P219M0qZ02EcPSoURmOBr4wFe+cVL12eqJHYys7zQg87+0EGbLqgGMJ+EwBylzVJgqFbZrZ6JMJcTZvE0Zgn68KSwgU97oSYZ1Q94AU+WyJ47DMFhQGTuDg4SPnOPeX/P0n5zOLBw7Cf8+nDsN263ptAXLJKPzNxOxkUJC3Q5aLvL2hEIadN+nzdYMI1N4mmHKBY+wzX0pi4O/CPeRy8V1g/B5/3EDko47WSfFf6+CAypjW4noodDUJMBblyKH/rfoMziBxSGHXWywoMU1scnEdEPBV09d+xP4DqgT7sgsv1OiMqMRSeCwvD6OA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=xagsolutions.com; dmarc=pass action=none header.from=xagsolutions.com; dkim=pass header.d=xagsolutions.com; arc=none
Received: from BYAPR06MB4022.namprd06.prod.outlook.com (2603:10b6:a02:8c::14) by BYAPR06MB5462.namprd06.prod.outlook.com (2603:10b6:a03:da::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.21; Wed, 18 Nov 2020 00:56:51 +0000
Received: from BYAPR06MB4022.namprd06.prod.outlook.com ([fe80::f801:8798:1298:c3a2]) by BYAPR06MB4022.namprd06.prod.outlook.com ([fe80::f801:8798:1298:c3a2%5]) with mapi id 15.20.3477.038; Wed, 18 Nov 2020 00:56:50 +0000
From: Sean Croghan <sean@xagsolutions.com>
To: "109all@ietf.org" <109all@ietf.org>
Thread-Topic: NOC update
Thread-Index: AQHWvUWzhUFWsj0s0E6voOldu/Wg1g==
Date: Wed, 18 Nov 2020 00:56:50 +0000
Message-ID: <1A8D2B8D-CC33-4430-B4FB-61995B3CF5EF@xagsolutions.com>
Reply-To: "109attendees@ietf.org" <109attendees@ietf.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-mailer: Apple Mail (2.3608.120.23.2.4)
authentication-results: ietf.org; dkim=none (message not signed) header.d=none;ietf.org; dmarc=none action=none header.from=xagsolutions.com;
x-originating-ip: [50.53.103.206]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 09322864-1670-404b-b87c-08d88b5cd587
x-ms-traffictypediagnostic: BYAPR06MB5462:
x-microsoft-antispam-prvs: <BYAPR06MB54627D8CC773A7BCE33E122BB8E10@BYAPR06MB5462.namprd06.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:9508;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: bNj/nWNCGGLee9Ml28tQONrcK+2pE5JXOQTQzMCypjTFc2L2vox5L2HOnN6l6bgxDSisayn6b1jjvThDpHWFVR18hU0b8as9m2YHuzC+aRS1O7IGSwGLVyVrnEmCcxxZ6EjGx3KmQu8zGrU6dwwmJSCpNLus0eADzyWwtXMYC5yPuFfDCOVt+1qns2s3WI8G4N6Pc3DqXj7yuw8TwS1cD5f1o4tjliplHRduQnfNspuYvCYlOqusZ/pViunQWjM9sV1IhBuNN5WJcQTPJ4NtnL9F0FBPJL2IS3TPacYVgJkQbY8rpNLiHP6nk2iAHzy+0jqygjfb9ZX8AP48B+epRg==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR06MB4022.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(346002)(366004)(15650500001)(8936002)(66556008)(76116006)(66446008)(66946007)(66476007)(64756008)(6916009)(186003)(33656002)(4743002)(91956017)(86362001)(8676002)(7116003)(2616005)(5660300002)(3480700007)(2906002)(6486002)(8796002)(71200400001)(83380400001)(66806009)(36756003)(508600001)(6512007)(26005)(6506007); DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata: ORxrdpxWRS8Yz0L3QitihjdtP2Zy/36gL4/CEOY0Fog2l00+uo6j+BtDRI2kL9eFoeOFlhQW8JKR+fsJl/WCiyjGXwxPqMBIJD3nnmqdGTsyXJBjsVWIpKBT2e8w0Q7+kgsoCEUQfxxJIhHCN5/zetzBA8eppiHhCHQBL0P/Ve8waP5pGvh3y74CccWoIYE1S2GCznAOIy7UpwMVfFE8FMHqdWsAR+LZjwDP4XXJrVP/45qEAodZV6M/ke/L2SnGbCmdYSlp1DnDrAcnyztQgoP3yP6esGKRHkitDhVIHcfkjxdGI1uMlvR3AMYE3rT9wKwhslRbVtDKPuaO2j7BVj+jDyiBsJxTqNHzTnqXx0cx174l+NaXUugq5tSSj2NTzCW8jwYaSjBderoJ0yChAstWdRcpxKjyAQaTJUhvbCAtm/r5AnqRpi3djNk4HIPg0JfffP9tL9xTrBgoXLlILJ8jXTKJDgfNRieurwbfbfgxpDAM3ZoLJ7eUiyqAJK1aDYN8hNNEcXC40ozqbx0qnyPg8lOYhizsiR2Mdf727i74mfWyuNJX8z3SjURzDcQyoRlCp95Vrk1CoWp4Mq53YqJt08KV8WRsLuenNSEaCwgpm2t6mjKkn8PsnJ5YiC08idpz6G2F5RmEpaShfZ10fg==
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_1A8D2B8DCC334430B4FB61995B3CF5EFxagsolutionscom_"
MIME-Version: 1.0
X-OriginatorOrg: xagsolutions.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BYAPR06MB4022.namprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 09322864-1670-404b-b87c-08d88b5cd587
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Nov 2020 00:56:50.7769 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 3905f813-0012-41b2-a403-d7a3a748e3c3
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: or8hT596mQcXgYafMraiz219TfpyIgoqDo/2xvOKutkP9liFnIDLQDeGlgYsk6tuUjyXlf5wfuOZqg3sm+hmiw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR06MB5462
Archived-At: <https://mailarchive.ietf.org/arch/msg/109all/YfHSIC5tuVZjSh4V78wUcE4peZ4>
Subject: [109all] NOC update
X-BeenThere: 109all@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Official communication about IETF 109 <109all.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/109all>, <mailto:109all-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/109all/>
List-Post: <mailto:109all@ietf.org>
List-Help: <mailto:109all-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/109all>, <mailto:109all-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 18 Nov 2020 00:56:56 -0000


I have an update for those of you affected by the outage in yesterdays IABOPEN session. We have isolated this to a interrupt to the virtual machines network interface. We currently have no explanation for this outage. We have engaged the hardware and network team with Azure to determine the cause of this event but do not have an explanation at this time.

I will provide an update when we have received more information.


For those interested in details:

At 07:56:36 UTC the network interface (eth0) went link down and the interface was removed from the VM
At 08:00:28 UTC then a new interface was added to the VM
At 08:00:29 UTC (eth1) went link up

Yes the VM added a new interface. The servers were provisioned with SR-IOV and we suspect that a migration event occurred that moved the VM to different hardware causing the NIC driver to be reloaded. We have found some evidence that would support our theory that a migration or unscheduled maintenance event occurred and are working to verify if that happened during this event. We have removed SR-IOV from the network interfaces on all servers.

I hope you are having a good and productive week


— The IEFT NOC Team