[109all] Network close and update

Everyone,

First an update to the network interruption during iabopen and the plenary. We have confirmed that backend maintenance events occurred during the outages. Azure Direct Support provided the following detail for the events.

We identified the occurrence was caused by an Azure initiated memory-preserving update action.
Note that RDP and SSH connections to the VM, or requests to any other services running inside the VM, could have failed during this time.
This update is part of routine maintenance performed on the underlying hosts for this VM. During these updates, the VM is frozen for up to 30 seconds and then resumed.
We apologize for any inconvenience this may have caused you. We are continuously working to improve the platform to reduce incidences of virtual machine unavailability.

For 110 we will have processes in place to mitigate/eliminate this issue.

So as we close this meeting I wish you all a safe walk to your bed and a quick adjustment back to your normal time. Also a big thank you to the NOC Team of Volunteers and Staff, a truly amazing group of people.

• Rick Alfvin (Linespeed)
• Alessandro Amirante (Meetecho)
• Hirochika Asai (Preferred Networks/WIDE)
• Rob Austein (Arrcus/DRL)
• Tobia Castaldi (Meetecho)
• Joe Clarke (Cisco)
• Bill Fenner (Arista)
• Bill Jensen (University of Wisconsin–Madison)
• Hans Kuhn (NSRC)
• Nick Kukich (Linespeed)
• Warren Kumari (Google)
• Lucy Lynch
• Lorenzo Miniero (Meetecho)
• Karen O'Donoghue (ISOC)
• Con Reilly (Linespeed)
• Simon Pietro Romano (Meetecho)
• Paolo Saviano (Meetecho)
• Clemens Schrimpe

I hope you all have a safe and wonderful New Year!

— Sean

On Nov 18, 2020, at 5:56 PM, Sean Croghan <sean@xagsolutions.com<mailto:sean@xagsolutions.com>> wrote:

As previously reported, we tracked down the cause of the interruption of the iabopen session to an issue with an unexpected Azure network interface removal event on network interfaces provisioned with SR-IOV.  To prevent this happening again we intended to remove SR-IOV networking entirely.  Unfortunately it now transpires that this change did not get applied to 2 of the 16 VMs including the application VM for the Plenary. So to add to the list of reasons to want 2020 to be over, towards the end of Plenary the same network interface removal event occurred and triggered an outage long enough to affect everyone.

I can confirm that the SR-IOV provisioning has now been removed from all VMs, which we believe eliminates the risk of the same thing happening again.  We continue to work with Azure Direct Support to determine the underlying cause of the removal events.

Please let me know if you have any questions.

Sean

On Nov 17, 2020, at 4:56 PM, Sean Croghan  wrote:

I have an update for those of you affected by the outage in yesterdays IABOPEN session. We have isolated this to a interrupt to the virtual machines network interface. We currently have no explanation for this outage. We have engaged the hardware and network team with Azure to determine the cause of this event but do not have an explanation at this time.

I will provide an update when we have received more information.

For those interested in details:

At 07:56:36 UTC the network interface (eth0) went link down and the interface was removed from the VM
At 08:00:28 UTC then a new interface was added to the VM
At 08:00:29 UTC (eth1) went link up

Yes the VM added a new interface. The servers were provisioned with SR-IOV and we suspect that a migration event occurred that moved the VM to different hardware causing the NIC driver to be reloaded. We have found some evidence that would support our theory that a migration or unscheduled maintenance event occurred and are working to verify if that happened during this event. We have removed SR-IOV from the network interfaces on all servers.

I hope you are having a good and productive week

— The IEFT NOC Team

--
109all mailing list
109all@ietf.org<mailto:109all@ietf.org>
https://www.ietf.org/mailman/listinfo/109all