rewindex

ChangesLookerMultiple services were experiencing issues in the europe-west2 region.

Looker: Incident updated: Multiple services were experiencing issues in the europe-west2 region.

Incident updateNot classified yetChanged status.cloud.google.com (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Discussed on Hacker News

Threads are matched automatically by vendor name and timing; points and comments refresh every 30 minutes. Method and limits.

Incident page: the full timeline of updates

Part of an outage event

Google Cloud europe-west2 outage, Oct 20, 2026 · 13 vendors, 13 incidents

Looker’s incident is the root of the event (the provider’s own post). (root, posted +0 min vs Google Cloud’s own post)

See the cascade chart and every vendor →

Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.

What changed

Struck in red is what Looker removed; blue is what they added. Muted text is unchanged context.

ChangedUpdateresolved → service_disruption#1
Before: **Summary** Multiple services were experiencing issues in the europe-west2 region. **Description** The issue with impacted GCP products has been mitigated for most of the customers and it is believed to be affecting currently a very small number of customers and our Engineering Team is working on it. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. No further updates will be provided here. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we're working on resolving the issue. **Symptoms** Google Cloud services were unavailable in europe-west2-b zone; some services in europe-west2 region might experienced elevated errors and intermittent network issues. **Workaround** None as the issue has been mitigated.
After: **Summary** Multiple services were experiencing issues in the europe-west2 region. **Description** The issue with impacted GCP products has been mitigated for most of the customers and it is believed to be affecting currently a very small number of customers and our Engineering Team is working on it. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. No further updates will be provided here. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we're working on resolving the issue. **Symptoms** Google Cloud services were unavailable in europe-west2-b zone; some services in europe-west2 region might experienced elevated errors and intermittent network issues. **Workaround** None as the issue has been mitigated.
AddedUpdateresolved#2
Before: not present
After: **Preliminary Incident Report** We recognize that Google Cloud services are critical to your organization, and we sincerely apologize for the disruption this incident may have caused to your business. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. **Date/Time of the Issue (All time US/Pacific)** Incident Start: 19 October 2026 23:00 Incident End: 20 October 2026 05:10 Duration: 6 hours, 10 minutes **Summary** On Monday, 19 October 2026, multiple Google Cloud Platform services in the europe-west2 region experienced service disruptions, elevated error rates, Application Programming Interface (API) timeouts, and processing latency for a duration of 6 hours and 10 minutes. The disruption was triggered by an electrical power interruption at the datacenter facility hosting availability zone europe-west2-b, which temporarily took physical host servers, network routing hardware, and storage systems offline. This subsequently caused secondary failover and queuing delays across unimpacted regional infrastructure. **Preliminary Root Cause** The incident was triggered when a power grid disruption occurred at the datacenter facility hosting availability zone europe-west2-b. To maintain continuous power during the disruption, automatic transfer to facility backup power systems was triggered immediately. The facility load was transferred to the Uninterruptible Power Supply (UPS) battery banks during the start up sequence of on-site emergency diesel generators. The Programmable Logic Controller (PLC) automation system (both primary and backup PLCs) failed to initiate the logical automatic start/transfer operations of the generators, preventing backup generators from assuming the load. When UPS battery reserves were depleted, physical host servers, network switching fabric, and local storage arrays dedicated to europe-west2-b lost power and went offline. The sequence of events leading to customer impact was as follows: - At 22:53 US/Pacific, an electrical power disruption occurred at the datacenter facility hosting availability zone europe-west2-b, and a PLC (Programmable Logic Controller) startup fault prevented facility emergency generators from assuming the electrical load. Facility load continued to be supported by UPS battery banks until the batteries were depleted, starting at 23:00 US/Pacific. - When the physical host servers, local storage infrastructure, and network aggregation hardware in europe-west2-b lost power, it caused instances, database replicas, and worker nodes in that zone to become immediately unreachable. - Regional traffic-routing mechanisms and client applications initiated automated failover toward available infrastructure in adjacent availability zones within the region. - Cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts, causing initial job submission stalls and API latency spikes. - The sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones, creating elevated admission queues, delayed database failover completions, and intermittent 500 and 503 API errors across the region. Google engineers have begun a full root cause analysis (RCA) and will provide additional information once it is available. **Remediation** Google engineers were alerted to the disruption via automated monitoring systems and probers at 23:08 US/Pacific on Monday, 19 October 2026, which detected elevated API error ratios, job submission timeouts, and loss of infrastructure reachability in europe-west2-b. Engineering teams immediately mobilized to isolate the affected infrastructure and commence recovery: - Traffic Isolation: Google engineers isolated the offline availability zone (europe-west2-b) by adjusting regional service routing away from the affected infrastructure, preventing cross-zone client requests from timing out against offline endpoints. - Workload Reprioritization and Resource Safeguards: Engineers adjusted scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations, reduce resource contention on execution slots, and relieve queue backlogs for regional workloads. - Power Grid Restoration: On-site facility engineers conducted graceful restoration of a stable utility power feed once utility supply was back, within 1 hour. Following this, they focused and completed a reset and tested the emergency backup generator automation controllers to restore facility power. - Progressive Infrastructure Re-activation: Google engineers systematically restarted core Google Cloud infrastructure and services. - Controlled Traffic Re-admission: Once system health and storage integrity were verified, engineers executed gradual, controlled traffic re-admission back to zone europe-west2-b allowing accumulated job backlogs to clear smoothly. By 05:10 US/Pacific on Tuesday, 20 October 2026, service availability had recovered for the vast majority of customer workloads, and resolution for most of the customers across Google Cloud Platform services in the region was confirmed. Following the recovery of a small tail of remaining isolated workloads, the closure notification for the incident was posted by 07:03 US/Pacific. **Description of Impact** Between Monday, 19 October 2026, 23:00 and Tuesday, 20 October 2026, 05:10 US/Pacific, customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 10 minutes. During the initial phase from 23:00 to 04:29 US/Pacific (duration 5 hours, 29 minutes), customers experienced elevated error rates, API timeouts, and processing delays until most Google Cloud Platform services recovered. Following platform service restoration, some customer workloads running on Compute Engine instances experienced a longer recovery tail. By 04:46 US/Pacific new Virtual Machine creation resumed and by 05:10 US/Pacific over 99% of Compute Engine Virtual Machines were in a healthy state. Any remaining customers that are experiencing Virtual Machine problems should file a customer support issue as per normal procedure. While the physical power outage was isolated strictly to datacenter facilities in europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. The primary causes were 1) instantaneous demand surges on unimpacted compute capacity and 2) synchronous control-plane validation dependencies across zonal boundaries. Affected products included: BigQuery, Looker, Google Compute Engine, Google Kubernetes Engine, Regional Persistent Disk, and Cloud SQL. These services experienced unexpected degradation and failover issues driven by sudden compute capacity exhaustion from surging demand, and control-plane API timeouts waiting on validations from the offline zone. Workloads and resources hosted entirely within europe-west2-b became unreachable. Data storage services with failover capability such as Cloud SQL, AlloyDB, Memorystore for Redis, and Cloud Bigtable failed over into nearby unimpacted zones during the europe-west2-b outage. Multi-zonal and regional services that spanned across the europe-west2 region experienced elevated latency, transient API errors (such as HTTP 500 and 503 errors), and task admission queuing as failover traffic concentrated onto unimpacted regional resources and cross-zone calls to offline components timed out. Storage write operations in the region experienced a transient error rate during the initial failover period, and regional query admission queues temporarily peaked before rapidly subsiding as capacity re-balanced. The following Google Cloud Platform services experienced impact: - AlloyDB for PostgreSQL - Apigee - BigQuery Data Transfer Service - Cloud Data Fusion - Cloud Filestore - Cloud Firestore - Cloud Load Balancing - Cloud Memorystore - Cloud NAT - Cloud Run - Cloud Spanner - Cloud Workstations - Datastream - Dialogflow CX - Google App Engine - Google BigQuery - Google Cloud Bigtable - Google Cloud Dataflow - Google Cloud Dataproc - Google Cloud NetApp Volumes - Google Cloud Pub/Sub - Google Cloud SQL - Google Cloud Storage (GCS) - Google Compute Engine (GCE) - Google Kubernetes Engine (GKE) - Google SecOps (Security Information and Event Management [SIEM] and Security Orchestration, Automation, and Response [SOAR]) - Looker (Google Cloud core) - Looker Studio - Managed Airflow (Google Cloud Composer) - Managed Service for Apache Kafka - Memorystore for Memcached - Memorystore for Redis - Memorystore for Redis Cluster - Memorystore for Valkey - Network Connectivity Center - Persistent Disk - Private Service Connect - Speech-to-Text - Virtual Private Cloud (VPC) - VMWare engine

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.

How the archive is checked, and what it cannot prove.

Rewindex captured this after the last checkpoint; the next run () commits it.

History of this incident

TodayThursday, October 22Thu, Oct 22

2 rewrites, 2 incident changes

Tuesday, October 20

9 incident changes

All Looker changesLooker profileMarkdownJSON