Companies/DigitalOcean/Incidents

DigitalOcean: Cloud Control Panel and Multiple services

Timeline of updates

1. resolved2023-01-29 15:00 UTC

   Postmortem The Incident ‌ Beginning at 04:58 UTC, January 29th, the DigitalOcean NYC3 data center region experienced an Internet connection outage. This prevented Droplets and Droplet-based services, App Platform, Load Balancers, and Spaces in that data center region from accessing the Internet during the outages. Additionally, users were unable to access the Cloud Control Panel and the DigitalOcean API during the outage and event processing failures were evident in all regions. Customers may have also seen some gaps in metrics and monitoring during the incident. This incident was triggered by a software bug in the border (edge) routers of the NYC3 region. ‌ Our engineers were able to detect and begin an incident response for this issue within minutes of the first signs of impact. We quickly determined the root cause and began remediation efforts. ‌ The root cause was discovered to be a bug in the operating system software of critical edge routing hardware, causing these routing systems to crash. The bug was triggered by interpretation of automatically propagated routes. Routers propagate routing information between redundant nodes as efficiently as possible to avoid errors. This behavior of standard routing protocols, coupled with the bug, caused clustered routers to crash nearly concurrently. ‌ Without the routing service, the edge routers quickly degraded and stopped serving traffic. This cut off the DigitalOcean network behind the edge routers in question from the rest of the Internet during these crashes. Besides impact to customers, the lack of external network access complicated efforts to troubleshoot and fix problems and spread secondary impacts to nearly all DigitalOcean services. The core problem occurred in three waves. The initial outage consisted of 16 minutes of Internet connectivity loss, followed by two 15-minute periods of network congestion, and finally 50 minutes of congestion, as traffic was redirected to perform software upgrades. ‌ To fix this issue, DigitalOcean engineers deployed emergency software upgrades to the impacted systems. These upgrades were the most time-consuming part of the response and, when completed, they resolved the bug in the impacted systems for the long term. Internet access at the expected performance levels was restored for Cloud, API, Droplets, Droplet-based services like Managed Kubernetes and Load Balancers, App Platform, Functions, Monitoring, and Spaces by the end of the emergency software upgrades. Some asynchronous customer-triggered events like power offs and resizes were stuck until DigitalOcean engineers completed manual cleanup of affected events after the incident. Timeline of Events ‌ UTC Timeline 04:58 - Both edge routers crash in NYC3, causing a complete outage for NYC3 for customer and internal services, as well as global impact to some services.05:05 - Internal incident response begins. 05:12 - Network Engineering team gains access to the edge routers in NYC3 and begins investigation. 05:13 - Network Engineering team sees the routing process crashed. 05:14 - Edge1 recovered and took traffic. Complete outage ends, congestion exists for customer and internal services. 05:21 - Multiple Engineering teams across DigitalOcean still reporting down alerts. Network Engineering team sees that memory is exhausted on Edge2. 05:27 - Multiple Engineering teams confirm alerts are clearing and that cloud.digitalocean.com is accessible again. 05:40 - Edge2 recovered after manual intervention caused by hardware failure. Congestion ends for customer and internal services. 06:06 - Public status page posted with information about the outage . 06:24 - Edge1 crashes for a second time, causing customers to be unable to access or experience latency with Droplets and Droplet-based services via public/private networks. 06:36 - Network Engineering team makes the decision to emergency upgrade software on Edge1 to a newer version to help mitigate the crashes. 06:41 - Edge1 recovers and is abl

From DigitalOcean’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on status.digitalocean.com (opens in new tab).

DigitalOcean profileRecent DigitalOcean incidentsAll outages on January 29, 2023DigitalOcean in January 2023MarkdownJSON