# imgix: Major incident: Elevated rendering errors
**Summary:** _summary pending_
## Part of an outage event
- [Google Cloud outage, Jun 12, 2025: 101 vendors, 107 incidents](https://rewindex.com/events/2025-06-12-google-cloud)
- This incident blames Google Cloud as the cause. (hop 1, posted +14 min vs the first vendor naming Google Cloud)
- Evidence: “…uncached requests. \# What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in a…”
```mermaid
flowchart LR
root["Google Cloud
own incident not archived"]:::inferred
h1["Within an hour
29 vendors, 28 blamed Google Cloud
Pinecone, Northflank, imgix +26"]:::blames
root --> h1
h4["1 to 4 hours later
12 vendors, 11 blamed Google Cloud
Smartsheet, Daily, Confluent +9"]:::blames
root --> h4
later["Later
1 vendor, 1 blamed Google Cloud
Zilliz"]:::blames
root --> later
p0["Firebase
own incident not archived"]:::inferred
root -.-> p0
k0["Codeium, Windsurf
+13 min"]:::hop2
p0 -->|"blamed Firebase"| k0
p1["Anthropic
posted +28 min, did not name Google Cloud"]:::timing
root -.-> p1
k1["GitHub, GitHub Copilot
+24 min"]:::hop2
p1 -->|"blamed Anthropic"| k1
p2["Cloudflare
own incident not archived"]:::inferred
root -.-> p2
k2["Doppler, Jupyter (Anaconda), Socket +4
+35 min to +3 h 41 min"]:::hop2
p2 -->|"blamed Cloudflare"| k2
me["imgix (this incident)
posted +14 min"]:::me
h1 --- me
```
_Detected automatically from incidents that started together and named the same provider; a link means one incident named another vendor, not a confirmed cause._
- Kind: Major incident · Label: major (vendor)
- Changed: 2025-06-12 18:09 UTC · Detected: 2026-10-19 18:53 UTC
- [Source: https://stspg.io/qm9g6cjzq2nl](https://stspg.io/qm9g6cjzq2nl)
- [Company: imgix](https://rewindex.com/company/imgix.md)
- Versions: before (none: first archived) → after incident\_version:760/ww7rgnbpkqg7@fe160df2d110
- Change 1 of 1 for this incident
Incident page (full timeline): [Elevated rendering errors](https://rewindex.com/company/imgix/incidents/ww7rgnbpkqg7.md)
## What changed
**1. Added: Title**
```diff
+ Elevated rendering errors
```
**2. Added: Impact**
```diff
+ major
```
**3. Added: Status**
```diff
+ postmortem
```
**4. Added: Start time**
```diff
+ 2025-06-12 18:09 UTC
```
**5. Added: Resolved time**
```diff
+ 2025-06-12 20:52 UTC
```
**6. Added: Affected components**
```diff
+ API Service, Rendering Infrastructure, Web Administration Tools
```
**7. Added: Update** (investigating) · 2025-06-12 18:09 UTC
```diff
+ We are investigating elevated render error rates for the service. Previously cached derivatives are not impacted.
```
**8. Added: Update** (identified) · 2025-06-12 18:15 UTC
```diff
+ The issue has been identified and we are investigating a solution.
```
**9. Added: Update** (identified) · 2025-06-12 18:53 UTC
```diff
+ The service is experiencing elevated error rates due to a major Google Cloud outage affecting services downstream. Previously cached derivatives are not impacted. We are investigating ways to mitigate this issue.
```
**10. Added: Update** (monitoring) · 2025-06-12 20:26 UTC
```diff
+ The service is restored. We are monitoring the situation.
```
**11. Added: Update** (identified) · 2025-06-12 20:38 UTC
```diff
+ The Rendering API has fully recovered. We are continuing to investigate Web Administration related issues (login and Management API) related to the incident.
```
**12. Added: Update** (monitoring) · 2025-06-12 20:47 UTC
```diff
+ The Rendering API is fully recovered. Web Administration tooling (logins and the Management API) are recovering. We are monitoring the results.
```
**13. Added: Update** (resolved) · 2025-06-12 20:52 UTC
```diff
+ The service is completely restored.
```
**14. Added: Update** (postmortem) · 2025-06-20 19:03 UTC
```diff
+ # Incident Summary Between **17:55 and 20:22 UTC** on **June 12, 2025**, Imgix services experienced major service disruptions across several key interfaces: * **Dashboard and Asset Manager**: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. * **Management API**: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. * **Rendering API:** Approximately 8% of all Imgix service requests failed due to a high error rate \(~80%\) for **uncached** assets via the Rendering API. Requests in the EU saw a lower failure rate \(~50%\) and a faster recovery time \(45 minutes\) for these uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the [Google Cloud outage here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1). # What happened * **17:55 UTC:** Internal alerts triggered due to a spike in rendering errors and service timeouts. * **18:01 UTC:** A short investigation uncovers several timeouts and increased error rates from Google Cloud. * **18:09 UTC:** Our status page is updated. * **18:53 UTC:** A major Google Cloud outage is confirmed, after which we update our status page. * **18:00–20:16 UTC:** Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. * **Throughout:** We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. * **20:16 UTC:** Google reported recovery in all regions except ʹus-central1ʹ. This allowed us to verify **significantly lower error rates for EU traffic** * **20:47 UTC:** Imgix systems achieved full recovery. * **20:52 UTC:** The incident was officially resolved on our status page. * **21:23 UTC:** Google confirms a full-service recovery at 21:23 UTC. # What went wrong * Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. * 3rd party services \(such as our CDN\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. * The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. # What we will do to prevent this in the future * Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. * Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. * Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency.
```
## History of this incident
| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-06-12 18:09 UTC | Major incident: Elevated rendering errors (this change) | major |
All imgix changes: [timeline](https://rewindex.com/company/imgix/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._
---
Canonical: https://rewindex.com/change/ica52d46349c6e3 · Markdown: https://rewindex.com/change/ica52d46349c6e3.md · JSON: https://rewindex.com/api/v1/changes/ica52d46349c6e3 · Generated 2026-10-20T00:48:52.833Z