Companies/DigitalOcean/Incidents

DigitalOcean: Cloud Control Panel and API

Timeline of updates

1. resolved2022-12-06 17:00 UTC

   Impact: critical; Our Engineering team has confirmed the full resolution of this issue, which impacted our Cloud Control Panel, API, Managed Kubernetes clusters, and Block Storage Volumes. Between 16:47 - 17:18 UTC, users experienced 500 errors and/or latency when attempting to reach the Cloud Control Panel and API. Between 16:47 - 18:49 UTC, users were unable to successfully attach/detach Block Storage Volumes. Between 16:47 - 20:50 UTC, users were unable to create new Managed Kubernetes clusters, as well as a subset of users were unable to destroy or upgrade existing clusters. If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Affected Components: Services - API Services - Block Storage Services - Cloud Control Panel Services - Kubernetes Regions - Global Status: resolved 2022-12-06 22:38 UTC - Our Engineering team has put further remediations in place and reconciliation efforts for Managed Kubernetes clusters have completed. At this time, creation for new clusters is enabled again and all cluster operations should be succeeding. With the root cause identified and remediations in place, we are no longer observing any impact to users. We'll continue to monitor the situation and post an update once we confirm the issue is fully resolved. Thank you for your patience. Affected Components: Services - API Services - Block Storage Services - Cloud Control Panel Services - Kubernetes Regions - Global Status: monitoring 2022-12-06 21:03 UTC - Our Engineering team continues work to remediate the root cause of this incident. Further changes have been made and Managed Kubernetes cluster creation will now fail outright should users attempt to provision a cluster. Attach and detach operations on Block Storage Volumes (except for Managed Kubernetes) are now available and are succeeding as expected. Affected Components: Services - API Services - Block Storage Services - Cloud Control Panel Services - Kubernetes Regions - Global Status: identified 2022-12-06 18:49 UTC - Our Engineering team has identified additional impact related to the root cause of this incident. At this time, creation, deletion, and upgrading of Managed Kubernetes clusters may result in failure due to a failing dependency of control planes. Attach and detach operations on Block Storage Volumes also remain unavailable. Affected Components: Services - API Services - Block Storage Services - Cloud Control Panel Services - Kubernetes Regions - Global Status: identified 2022-12-06 17:58 UTC - As a result of necessary remediation steps, attach and detach operations on Block Storage Volumes in all regions are currently unavailable. Our Engineering team is continuing work to fully remediate the root cause. The Cloud Control Panel and API remain accessible. Affected Components: Services - API Services - Block Storage Services - Cloud Control Panel Regions - Global Status: identified 2022-12-06 17:40 UTC - The implemented actions have been successful in remediating the impact of this incident and our Engineering team has confirmed error rates are returning to normal thresholds. At this time, the Cloud Control Panel and API are accessible again. Affected Components: Services - API Services - Cloud Control Panel Regions - Global Status: identified 2022-12-06 17:25 UTC - Our Engineering team has identified the cause of the issue with the Cloud Control Panel and API and is now taking steps to mitigate impact. We're seeing some recovery, but users may still be unable to reach the Control Panel or API. We'll post an update once we confirm the implemented actions have remediated impact. Affected Components: Services - API Services - Cloud Control Panel Regions - Global Status: investigating 2022-12-06 17:14 UTC - We are investigating reports of issues reaching the DigitalOcean Cloud Control Panel and API. At this time, users may see 500 errors or latency when attempting to reach both the Control Panel and API. We'll pr

From DigitalOcean’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on status.digitalocean.com (opens in new tab).

DigitalOcean profileRecent DigitalOcean incidentsAll outages on December 6, 2022DigitalOcean in December 2022MarkdownJSON