Rewindex

RewindexOutage eventsDec 7, 2021

AWS us-east-1 outage, Dec 7, 2021

71 vendors posted 76 incidents between 15:00 on Dec 7 and 03:00 on Dec 8 UTC, against about 17 incidents from 14 vendors in the same hours on an ordinary week. 28 vendors named AWS in 29 incidents (22 of them blamed it). Vendors reported an AWS outage; AWS’s own incident for this date is not in the archive.

Observed, not verified. A line means a vendor’s incident named another vendor, or started in the same window. It is not a proven cause.

Times are UTC; lags are from when the first vendor named AWS.

    25. first vendor named AWS 15:50
Named AWSNamed another vendorDid not name anyoneEach column is 30 minutes

How it spread

Vendors that named AWS, grouped by when they posted, and vendors that blamed one of the vendors AWS took down. Solid lines: the vendor named the other. Dashed: same window, not named.

Swipe sideways to see the whole diagram.

AWS us-east-1own incident not archivedWithin an hour11 vendors, 9 blamed AWSAptible, Firebolt, Honeycomb, Asana +71 to 4 hours later16 vendors, 12 blamed AWSMaven Central, Pusher, Confluent, JFrog +12Later3 vendors, 2 blamed AWSSegment, Confluent, Doppler
AWS us-east-1own incident not archivedWithin an hour11 vendors, 9 blamed AWSAptible, Firebolt, Honeycomb,Asana +71 to 4 hours later16 vendors, 12 blamed AWSMaven Central, Pusher,Confluent, JFrog +12Later3 vendors, 2 blamed AWSSegment, Confluent, Doppler
Rendered on the server with pele from the Mermaid source below. Each box lists the first few vendors; every vendor is listed further down.
Mermaid source
flowchart LR
  root["AWS us-east-1<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>11 vendors, 9 blamed AWS<br/>Aptible, Firebolt, Honeycomb, Asana +7"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>16 vendors, 12 blamed AWS<br/>Maven Central, Pusher, Confluent, JFrog +12"]:::blames
  root --> h4
  later["Later<br/>3 vendors, 2 blamed AWS<br/>Segment, Confluent, Doppler"]:::blames
  root --> later

Vendors that named AWS

In their own words. “Blames” means a causal phrase sits next to the name (“due to”, “upstream provider”…).

Within an hour

11 vendors

  • +0 min

    Blames AWS

    Major incident

    AptibleOperations suspended due to AWS outage

    …Operations suspended due to AWS outage We have enabled all operations platform-wide, in all regions, but will continue to monitor for any regression.…

  • +8 min

    Blames AWS

    Major incident

    FireboltEngines may fail to start and stop

    …ngines are fully operational. Engines may fail to start and stop due to partial issues on AWS us-east-1 region. The incident has been resolved. Engines are fully operational. The issue has been identi…

  • +9 min

    Blames AWS

    Major incident

    HoneycombQuerying and alerting outage due to underlying cloud provider failure

    …Querying and alerting outage due to underlying cloud provider failure Our underlying AWS us-east-1 dependencies have recovered and all systems at Honeycomb continue to be stable.…

  • +19 min

    Blames AWS

    Minor incident

    AsanaAWS Outage

    …AWS Outage We have mostly recovered and our API and Event Distribution should be catching up soon.…

  • +30 min

    Blames AWS

    Major incident

    UnbounceCannot Publish, Unpublish or Republish Pages

    …Cannot Publish, Unpublish or Republish Pages This incident has been resolved. Amazon Web Services has resolved the issue, and their affected services have been restored. As a result, Unbounce's servic…

  • +39 min

    Blames AWS

    Major incident

    GrainIncreased error rates from Amazon Web Services

    …Increased error rates from Amazon Web Services This incident has been resolved.…

  • +46 min

    Blames AWS

    Major incident

    KeeperAWS Outage - US-EAST Affected

    …AWS Outage - US-EAST Affected AWS has resolved their API errors. We will continue to monitor the situation and post any updates on the Keeper status page.…

  • +47 min

    Blames AWS

    Major incident

    MattermostUnable to create new workspaces

    …for any further issues by the cloud provider. There is an outage in the cloud provider in us-east-1 We are continuing to monitor for any further issues.…

  • +49 min

    Names AWS

    Major incident

    SnowflakeAWS - US East (N. Virginia): MI-20211207

    …AWS - US East (N. Virginia): MI-20211207 The Service Incident is now closed. A Root Cause Analysis (RCA) will be posted within the next ten business days. We apologize for the inconvenience caused by…

  • +51 min

    Names AWS

    Critical incident

    HeapAWS us-east-1

    …AWS us-east-1 This incident has been resolved.…

  • +56 min

    Blames AWS

    Critical incident

    PipedreamAWS incident affecting Pipedream

    …AWS incident affecting Pipedream This incident should be resolved.…

1 to 4 hours later

15 vendors

Later

3 vendors

Same window, did not name anyone

47 incidents started during the event without naming AWS or another vendor. Some are part of it, some are a normal day’s background (about 17 in these hours).

Show all 47

Detected automatically from 33 status pages opening incidents in the peak hour (normally at most 5). Mentions are matched by name in each incident’s title and updates, then classified as cause, location or integration; quoted text is the vendor’s. Vendors that share one status page are shown once.