Rewindex

RewindexOutage eventsSep 28, 2023

AWS outage, Sep 28, 2023

23 vendors posted 24 incidents between 18:00 on Sep 28 and 04:00 on Sep 29 UTC, against about 17 incidents from 14 vendors in the same hours on an ordinary week. 9 vendors named AWS in 10 incidents (5 of them blamed it). Vendors reported an AWS outage; AWS’s own incident for this date is not in the archive.

Observed, not verified. A line means a vendor’s incident named another vendor, or started in the same window. It is not a proven cause.

Times are UTC; lags are from when the first vendor named AWS.

    21. first vendor named AWS 18:50
Named AWSNamed another vendorDid not name anyoneEach column is 30 minutes

How it spread

Vendors that named AWS, grouped by when they posted, and vendors that blamed one of the vendors AWS took down. Solid lines: the vendor named the other. Dashed: same window, not named.

Swipe sideways to see the whole diagram.

AWSown incident not archivedWithin an hour3 vendors, 1 blamed AWSAbly, Temporal Cloud, Keeper1 to 4 hours later4 vendors, 3 blamed AWSDremio, Firebolt, CockroachDB, Elastic CloudLater3 vendors, 1 blamed AWSAptible, Temporal Cloud, Confluent
AWSown incident not archivedWithin an hour3 vendors, 1 blamed AWSAbly, Temporal Cloud, Keeper1 to 4 hours later4 vendors, 3 blamed AWSDremio, Firebolt, CockroachDB,Elastic CloudLater3 vendors, 1 blamed AWSAptible, Temporal Cloud,Confluent
Rendered on the server with pele from the Mermaid source below. Each box lists the first few vendors; every vendor is listed further down.
Mermaid source
flowchart LR
  root["AWS<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>3 vendors, 1 blamed AWS<br/>Ably, Temporal Cloud, Keeper"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>4 vendors, 3 blamed AWS<br/>Dremio, Firebolt, CockroachDB, Elastic Cloud"]:::blames
  root --> h4
  later["Later<br/>3 vendors, 1 blamed AWS<br/>Aptible, Temporal Cloud, Confluent"]:::blames
  root --> later

Vendors that named AWS

In their own words. “Blames” means a causal phrase sits next to the name (“due to”, “upstream provider”…).

Within an hour

3 vendors

1 to 4 hours later

4 vendors

  • +1 h 16 min

    Blames AWS

    Minor incident

    DremioEngine degradation in AWS US-EAST-1 region

    …Engine degradation in AWS US-EAST-1 region AWS has resolved the issues with service in US-EAST-1. Dremio Cloud Engines (EC2 instances) should be fully operational again across all regions. We will con…

  • +1 h 21 min

    Blames AWS

    Major incident

    FireboltEngine Startup Issue in US-EAST-1 Region

    …resolved and the service is operating normally As of now, we are awaiting an update from AWS to resolve the engine startup problem. While we await AWS's response, it's important to note that the issu…

  • +2 h 20 min

    Blames AWS

    Minor incident

    CockroachDBAWS is experiencing networking issues in the us-east-1 region

    …AWS is experiencing networking issues in the us-east-1 region This incident has been resolved. This incident has been resolved. We are continuing to monitor for any further issues. We've been advised…

  • +3 h 12 min

    Names AWS

    Minor incident

    Elastic CloudAWS EC2 Network Issues in us-east-1 region

    …AWS EC2 Network Issues in us-east-1 region The AWS network connectivity has been restored on the us-east-1 region (use1-az2 zone). Everything is operating normally in the region.…

Later

3 vendors

Same window, did not name anyone

14 incidents started during the event without naming AWS or another vendor. Some are part of it, some are a normal day’s background (about 17 in these hours).

Show all 14

Detected automatically from 8 status pages opening incidents in the peak hour (normally at most 6). Mentions are matched by name in each incident’s title and updates, then classified as cause, location or integration; quoted text is the vendor’s. Vendors that share one status page are shown once.