Associated Students Of Information Technology
[INFO] Service Outage Alert: OpenStack is down [RESOLVED] - Printable Version

+- Associated Students Of Information Technology (http://corvallisitclub.com)
+-- Forum: Club Works (http://corvallisitclub.com/forumdisplay.php?fid=7)
+--- Forum: Announcements (http://corvallisitclub.com/forumdisplay.php?fid=11)
+--- Thread: [INFO] Service Outage Alert: OpenStack is down [RESOLVED] (/showthread.php?tid=95)



Service Outage Alert: OpenStack is down [RESOLVED] - ITZach - 01-06-2018

As of 6pm January 6th we are aware of a total service outage on the OpenStack cluster. Services are not responding at the gateway so no traffic is getting in or out of the cluster. At this time we do not have a known cause for this outage and we will be working with COE IT in the coming days to isolate and fix the problem. As we do not have direct access to these systems it will take time to resolve the outage. Thank you for your patience and understanding. Updates will be posted here as they become available.

UPDATE 1/9/18 05:50: At 2pm yesterday physical access was granted and we were able to reboot the gateway. Unfortunately the gateway was not the only issue and the servers are still not responding to requests. We will update when we know more.

UPDATE 1/9/18 09:56: As of 9am this morning the OpenStack cluster is functioning normally. After regaining access to the datacenter this morning we were able to determine that the servers did not obtain the correct IP information from the gateway after it was rebooted. An emergency reboot of the cluster was sufficient to correct the issue. We will be looking into why the services did not come back online after the gateway reboot and until we find an answer we suspect the issue may occur again. We will also be working to replace the hardware that the gateway is running on to improve performance and protect against further interruptions of service.