Changes / Firebolt / Unable to start Engines on FB 2.0

Firebolt: Critical incident: Unable to start Engines on FB 2.0

Summary

Summary pending. The diff below shows exactly what changed.

Major incidentLabel: outage (model (Jev), 100%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: Unable to start Engines on FB 2.0
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2024-11-14 19:46 UTC
AddedResolved time#5
Before: not present
After: 2024-11-14 20:10 UTC
AddedAffected components#6
Before: not present
After: Engines, Firebolt Components - Engines
AddedUpdateinvestigating#7
Before: not present
After: We are currently experiencing an unplanned service outage affecting all regions. As of 18:26 UTC, you may encounter difficulties starting engines in our FB 2.0 environment. Our engineering team is actively working to resolve the issue as quickly as possible. We will provide updates on our Statuspage at least every 15 minutes. Thank you for your patience and understanding.
AddedUpdateinvestigating#8
Before: not present
After: We are still experiencing the issue, and continuing to work on a fix Updated Impact: - Engines are unable to start. - Engines are unable to stop with Autostop and remain in the "Running" state, even when no queries are executed. - Insert commands are failing on currently running engines. We continue to look into this with the highest priority.
AddedUpdateresolved#9
Before: not present
After: The issue was resolved at 20:03 UTC. We experienced an outage in engine operations, which was addressed by reverting a recent network change. Services are now fully operational. Please reach out if you experience any further issues.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Critical incident: Unable to start Engines on FB 2.0 (this change)Major incident

All Firebolt changes → · Firebolt profile · Markdown · JSON