Changes / Firebolt / Engine Startup Issue in US-EAST-1 Region

Firebolt: Major incident: Engine Startup Issue in US-EAST-1 Region

Summary

Summary pending. The diff below shows exactly what changed.

Major incidentLabel: outage (model (Jev), 99%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: Engine Startup Issue in US-EAST-1 Region
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2023-09-28 20:10 UTC
AddedResolved time#5
Before: not present
After: 2023-09-29 06:37 UTC
AddedAffected components#6
Before: not present
After: AWS - AWS ec2-us-east-1, AWS ec2-us-east-1
AddedUpdateinvestigating#7
Before: not present
After: AWS experience an issue in US-EAST-1 region that may affect the startup of engines. This issue primarily impacts the launch of new instances within this specific region. Our team is actively monitoring the situation, and we are in close communication with AWS to expedite the resolution of this issue. We will provide updates as we receive more information or make progress toward a solution. You can also track it on the Status page of AWS - https://health.aws.amazon.com/health/status
AddedUpdateinvestigating#8
Before: not present
After: AWS experience an issue in US-EAST-1 region that may affect the startup of engines. This issue primarily impacts the launch of new instances within this specific region. Our team is actively monitoring the situation, and we are in close communication with AWS to expedite the resolution of this issue. We will provide updates as we receive more information or make progress toward a solution. You can also track it on the Status page of AWS - https://health.aws.amazon.com/health/status
AddedUpdateidentified#9
Before: not present
After: As of now, we are awaiting an update from AWS to resolve the engine startup problem. While we await AWS's response, it's important to note that the issue continues to affect the launch of new instances in the US-EAST-1 region. However, our services in other regions as well as currently running engines remain unaffected and are functioning as expected. Please continue to follow updates here and in AWS status page - https://health.aws.amazon.com/health/status
AddedUpdatemonitoring#10
Before: not present
After: While the AWS outage is not fully resolved, it is gradually recovering, and so do our engines. We continue to monitor our affected engines until we gain a fully operational status. For more updates on the AWS status, please see here: https://health.aws.amazon.com/health/status
AddedUpdatemonitoring#11
Before: not present
After: No significant updates have been provided from AWS. We are monitoring their status page as well as our internal monitoring tools. We do no expect to provide further interim updates, we will update here once the issue is resolved.
AddedUpdateresolved#12
Before: not present
After: The issue has been resolved and the service is operating normally

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Major incident: Engine Startup Issue in US-EAST-1 Region (this change)Major incident

All Firebolt changes → · Firebolt profile · Markdown · JSON