# imgix: Elevated rendering errors
- Impact (vendor’s label): major
- Status: postmortem
- Started: 2025-06-12 18:09 UTC · Resolved: 2025-06-12 20:52 UTC · Duration: 2.7 h
- Updates: 8 · Affected components: API Service, Rendering Infrastructure, Web Administration Tools
- [Vendor status page: https://stspg.io/qm9g6cjzq2nl](https://stspg.io/qm9g6cjzq2nl)
- [Company: imgix](https://rewindex.com/company/imgix.md)
## Part of an outage event
- [Google Cloud outage, Jun 12, 2025: 101 vendors, 107 incidents](https://rewindex.com/events/2025-06-12-google-cloud)
- This incident blames Google Cloud as the cause. (hop 1, posted +14 min vs the first vendor naming Google Cloud)
- Evidence: “…uncached requests. \# What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in a…”
```mermaid
flowchart LR
root["Google Cloud
own incident not archived"]:::inferred
h1["Within an hour
29 vendors, 28 blamed Google Cloud
Pinecone, Northflank, imgix +26"]:::blames
root --> h1
h4["1 to 4 hours later
12 vendors, 11 blamed Google Cloud
Smartsheet, Daily, Confluent +9"]:::blames
root --> h4
later["Later
1 vendor, 1 blamed Google Cloud
Zilliz"]:::blames
root --> later
p0["Firebase
own incident not archived"]:::inferred
root -.-> p0
k0["Codeium, Windsurf
+13 min"]:::hop2
p0 -->|"blamed Firebase"| k0
p1["Anthropic
posted +28 min, did not name Google Cloud"]:::timing
root -.-> p1
k1["GitHub, GitHub Copilot
+24 min"]:::hop2
p1 -->|"blamed Anthropic"| k1
p2["Cloudflare
own incident not archived"]:::inferred
root -.-> p2
k2["Doppler, Jupyter (Anaconda), Socket +4
+35 min to +3 h 41 min"]:::hop2
p2 -->|"blamed Cloudflare"| k2
me["imgix (this incident)
posted +14 min"]:::me
h1 --- me
```
_Detected automatically from incidents that started together and named the same provider; a link means one incident named another vendor, not a confirmed cause._
## Timeline of updates
- **investigating** 2025-06-12 18:09 UTC: We are investigating elevated render error rates for the service. Previously cached derivatives are not impacted.
- **identified** 2025-06-12 18:15 UTC: The issue has been identified and we are investigating a solution.
- **identified** 2025-06-12 18:53 UTC: The service is experiencing elevated error rates due to a major Google Cloud outage affecting services downstream. Previously cached derivatives are not impacted. We are investigating ways to mitigate this issue.
- **monitoring** 2025-06-12 20:26 UTC: The service is restored. We are monitoring the situation.
- **identified** 2025-06-12 20:38 UTC: The Rendering API has fully recovered. We are continuing to investigate Web Administration related issues (login and Management API) related to the incident.
- **monitoring** 2025-06-12 20:47 UTC: The Rendering API is fully recovered. Web Administration tooling (logins and the Management API) are recovering. We are monitoring the results.
- **resolved** 2025-06-12 20:52 UTC: The service is completely restored.
- **postmortem** 2025-06-20 19:03 UTC: \# Incident Summary Between \*\*17:55 and 20:22 UTC\*\* on \*\*June 12, 2025\*\*, Imgix services experienced major service disruptions across several key interfaces: \* \*\*Dashboard and Asset Manager\*\*: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. \* \*\*Management API\*\*: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. \* \*\*Rendering API:\*\* Approximately 8% of all Imgix service requests failed due to a high error rate \\(~80%\\) for \*\*uncached\*\* assets via the Rendering API. Requests in the EU saw a lower failure rate \\(~50%\\) and a faster recovery time \\(45 minutes\\) for these uncached requests. \# What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the \[Google Cloud outage here\](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW\#RP1d9aZLNFZEJmTBk8e1). \# What happened \* \*\*17:55 UTC:\*\* Internal alerts triggered due to a spike in rendering errors and service timeouts. \* \*\*18:01 UTC:\*\* A short investigation uncovers several timeouts and increased error rates from Google Cloud. \* \*\*18:09 UTC:\*\* Our status page is updated. \* \*\*18:53 UTC:\*\* A major Google Cloud outage is confirmed, after which we update our status page. \* \*\*18:00–20:16 UTC:\*\* Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. \* \*\*Throughout:\*\* We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. \* \*\*20:16 UTC:\*\* Google reported recovery in all regions except \`us-central1\`. This allowed us to verify \*\*significantly lower error rates for EU traffic\*\* \* \*\*20:47 UTC:\*\* Imgix systems achieved full recovery. \* \*\*20:52 UTC:\*\* The incident was officially resolved on our status page. \* \*\*21:23 UTC:\*\* Google confirms a full-service recovery at 21:23 UTC. \# What went wrong \* Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. \* 3rd party services \\(such as our CDN\\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. \* The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. \# What we will do to prevent this in the future \* Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. \* Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. \* Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency.
## Rewindex changes for this incident
- 2025-06-12 18:09 UTC: [Major incident: Elevated rendering errors](https://rewindex.com/change/ica52d46349c6e3.md)
---
Canonical: https://rewindex.com/company/imgix/incidents/ww7rgnbpkqg7 · Markdown: https://rewindex.com/company/imgix/incidents/ww7rgnbpkqg7.md · JSON: https://rewindex.com/api/v1/companies/imgix/incidents/ww7rgnbpkqg7 · Generated 2026-10-20T00:07:17.749Z