imgix: Elevated rendering errors
Google Cloud outage, Jun 12, 2025 · 101 vendors, 107 incidents
imgix’s incident blames Google Cloud as the cause. (hop 1, posted +14 min vs the first vendor naming Google Cloud)
“…uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in a…”
Swipe sideways to see the whole diagram.
Mermaid source
flowchart LR
root["Google Cloud<br/>own incident not archived"]:::inferred
h1["Within an hour<br/>29 vendors, 28 blamed Google Cloud<br/>Pinecone, Northflank, imgix +26"]:::blames
root --> h1
h4["1 to 4 hours later<br/>12 vendors, 11 blamed Google Cloud<br/>Smartsheet, Daily, Confluent +9"]:::blames
root --> h4
later["Later<br/>1 vendor, 1 blamed Google Cloud<br/>Zilliz"]:::blames
root --> later
p0["Firebase<br/>own incident not archived"]:::inferred
root -.-> p0
k0["Codeium, Windsurf<br/>+13 min"]:::hop2
p0 -->|"blamed Firebase"| k0
p1["Anthropic<br/>posted +28 min, did not name Google Cloud"]:::timing
root -.-> p1
k1["GitHub, GitHub Copilot<br/>+24 min"]:::hop2
p1 -->|"blamed Anthropic"| k1
p2["Cloudflare<br/>own incident not archived"]:::inferred
root -.-> p2
k2["Doppler, Jupyter (Anaconda), Socket +4<br/>+35 min to +3 h 41 min"]:::hop2
p2 -->|"blamed Cloudflare"| k2
me["imgix (this incident)<br/>posted +14 min"]:::me
h1 --- meFull event: timeline, evidence and all vendors →
Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.
Affected components: API ServiceRendering InfrastructureWeb Administration Tools
Timeline of updates
- investigating
We are investigating elevated render error rates for the service. Previously cached derivatives are not impacted.
- identified
The issue has been identified and we are investigating a solution.
- identified
The service is experiencing elevated error rates due to a major Google Cloud outage affecting services downstream. Previously cached derivatives are not impacted. We are investigating ways to mitigate this issue.
- monitoring
The service is restored. We are monitoring the situation.
- identified
The Rendering API has fully recovered. We are continuing to investigate Web Administration related issues (login and Management API) related to the incident.
- monitoring
The Rendering API is fully recovered. Web Administration tooling (logins and the Management API) are recovering. We are monitoring the results.
- resolved
The service is completely restored.
- postmortem
# Incident Summary Between **17:55 and 20:22 UTC** on **June 12, 2025**, Imgix services experienced major service disruptions across several key interfaces: * **Dashboard and Asset Manager**: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. * **Management API**: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. * **Rendering API:** Approximately 8% of all Imgix service requests failed due to a high error rate \(~80%\) for **uncached** assets via the Rendering API. Requests in the EU saw a lower failure rate \(~50%\) and a faster recovery time \(45 minutes\) for these uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the [Google Cloud outage here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1). # What happened * **17:55 UTC:** Internal alerts triggered due to a spike in rendering errors and service timeouts. * **18:01 UTC:** A short investigation uncovers several timeouts and increased error rates from Google Cloud. * **18:09 UTC:** Our status page is updated. * **18:53 UTC:** A major Google Cloud outage is confirmed, after which we update our status page. * **18:00–20:16 UTC:** Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. * **Throughout:** We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. * **20:16 UTC:** Google reported recovery in all regions except `us-central1`. This allowed us to verify **significantly lower error rates for EU traffic** * **20:47 UTC:** Imgix systems achieved full recovery. * **20:52 UTC:** The incident was officially resolved on our status page. * **21:23 UTC:** Google confirms a full-service recovery at 21:23 UTC. # What went wrong * Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. * 3rd party services \(such as our CDN\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. * The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. # What we will do to prevent this in the future * Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. * Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. * Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency.
Changes Rewindex caught on this incident
- Jun 12, 2025Major incident: Elevated rendering errors
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).