ChangesimgixElevated rendering errors
imgix: Critical incident: Elevated rendering errors
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what imgix removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Elevated rendering errors
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2024-06-17 00:21 UTC
Before: not present
After: 2024-06-17 01:19 UTC
Before: not present
After: Rendering Infrastructure
Before: not present
After: We are currently investigating elevated render error rates for uncached derivative images. We will update once when we obtain more information.
Previously cached derivatives are not impacted.
Before: not present
After: The issue has been identified and our engineering team is developing a fix.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: A fix has been implemented and error rates are returning to normal. We are continuing to monitor the service.
Before: not present
After: This incident has been resolved.
Before: not present
After: ## What happened?
On June 17, 2024, at 00:00 UTC, imgix experienced an extreme spike in requests to our render stack. This unexpected surge caused a failure in our auto-scaling infrastructure, leading to an inability to manage all incoming traffic effectively. A fix was implemented at 00:38 UTC, and the issue was resolved by 01:06 UTC.
## How were customers impacted?
Between 00:00 and 01:06 UTC, customers may have experienced failures when requesting new renders. However, previously cached assets served successfully during this time.
## What went wrong during the incident?
The incident was triggered by a significant increase in requests, which our automated systems did not properly handle. Although the system started to auto-scale as expected, the unexpected surge caused issues with the health checks used for auto-scaling. The combination of extra traffic and health check failure led to an inability to render new images that required manual intervention to resolve.
## What will imgix do to prevent this in the future?
To avoid similar incidents in the future, imgix is taking the following actions:
1. **Health Check Enhancement:** We have investigated and implemented updated health checks to support increased traffic volumes.
2. **Rate Limiting:** Further rate limits will be applied to manage traffic spikes and minimize their impact.
3. **Traffic Routing:** Traffic will be rerouted as necessary to distribute the load and reduce the risk of system overloads.
4. **Automated Alerts Improvement:** We will enhance our automated alert systems to respond more effectively to traffic surges and potential issues, including health check failures.
By addressing these areas, we aim to further improve our system's resilience and ensure a smoother customer experience during periods of high demand.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Monday, June 17, 2024
1 incident change
- CriticalCritical incident
Elevated rendering errorsthis change