# Firebolt: Engines may fail to start and stop

- Impact (vendor’s label): major
- Status: resolved
- Started: 2022-01-26 13:02 UTC · Resolved: 2022-01-26 16:48 UTC · Duration: 3.8 h
- Updates: 12 · Affected components: Engines, Firebolt Components - Engines
- [Vendor status page: https://stspg.io/rn2c126d09n4](https://stspg.io/rn2c126d09n4)
- [Company: Firebolt](https://rewindex.com/company/firebolt.md)

## Summary

Firebolt reported a major incident affecting engine starts and stops from 2022-01-26 13:02 UTC to 2022-01-26 16:48 UTC (3 h 45 min). Ingestion on impacted engines may also have been affected. Firebolt reported a master-cluster outage that disrupted internal services; a fix was deployed and tested before resolution. _(machine-written, sol)_
## Timeline of updates

- **investigating** 2022-01-26 13:02 UTC: Engines may fail to start and stop.
- **investigating** 2022-01-26 13:17 UTC: We are continuing to investigate this issue
- **identified** 2022-01-26 13:19 UTC: The issue has been identified, and a fix is being implemented.
- **identified** 2022-01-26 13:44 UTC: We are continuing to work on a fix for the issue.
- **identified** 2022-01-26 13:54 UTC: The issue is identified and fix being implemented. The issue affects only engines under US-EAST-1 region.
- **identified** 2022-01-26 14:16 UTC: We are still experiencing the issue, and continuing to work on a fix.
- **identified** 2022-01-26 15:28 UTC: We have fixed a portion of the issue with our engines and are now working on a complete fix.  This issue still only affects engines on US-EAST-1. Starting/Stopping engines in that region will not work until we resolve the issue. We will update you in 30 minutes or earlier if fixed.
- **identified** 2022-01-26 15:41 UTC: We have identified and are deploying a fix, we believe the process should take 30minutes - 1hour and we'll update you at the 30m mark if it is still running or complete.
- **identified** 2022-01-26 16:04 UTC: Our fix is still underway and looking good. We believe 15-30 more minutes until the issue is resolved. Please do not run any ingestion during this time. We will update you shortly once complete.
- **monitoring** 2022-01-26 16:20 UTC: The fix has been deployed successfully and we are now testing it to ensure the efficacy. This should take ~10 minutes Please note we found that any ingestion on the impacted engines during the timeframe may have been affected as well.  We are similarly testing those findings, and will update on what actions need to be taken if any.
- **monitoring** 2022-01-26 16:43 UTC: Testing is complete and the issue is resolved. A full update on the issue is below, please note the restoration time, and let us know if you notice any residual impacts to your systems. At approximately 1230 UTC we experienced an outage in our master cluster. Though we quickly rectified this particular issue, it had disrupted several internal services - most critically, one of our most important storage services entered an unresponsive state.  We made several attempts to regain the health of this system but in the end had to revert to a full restoration, which was successful. That restoration is timestamped at 12:07 UTC, meaning any ingestion done after that time will have been lost.
- **resolved** 2022-01-26 16:48 UTC: This incident has been resolved.

## Rewindex changes for this incident

- 2022-01-26 13:02 UTC: [Major incident: Engines may fail to start and stop](https://rewindex.com/change/i328d9694e34823.md)


---
Canonical: https://rewindex.com/company/firebolt/incidents/2p0m5hnrjq8t · Markdown: https://rewindex.com/company/firebolt/incidents/2p0m5hnrjq8t.md · JSON: https://rewindex.com/api/v1/companies/firebolt/incidents/2p0m5hnrjq8t · Generated 2026-10-18T09:45:59.927Z