Rewindex

RewindexOutage eventsNov 25, 2020

AWS outage, Nov 25, 2020

28 vendors posted 28 incidents between 13:00 on Nov 25 and 01:00 on Nov 26 UTC, against about 19 incidents from 15 vendors in the same hours on an ordinary week. 6 vendors named AWS in 7 incidents (6 of them blamed it). Vendors reported an AWS outage; AWS’s own incident for this date is not in the archive.

Observed, not verified. A line means a vendor’s incident named another vendor, or started in the same window. It is not a proven cause.

Times are UTC; lags are from when the first vendor named AWS.

    25. first vendor named AWS 15:16
Named AWSNamed another vendorDid not name anyoneEach column is 30 minutes

How it spread

Vendors that named AWS, grouped by when they posted, and vendors that blamed one of the vendors AWS took down. Solid lines: the vendor named the other. Dashed: same window, not named.

Swipe sideways to see the whole diagram.

AWSown incident not archivedWithin an hour1 vendor, 1 blamed AWSPipedream1 to 4 hours later4 vendors, 4 blamed AWSHubSpot, Xero, Lattice, HeapLater2 vendors, 2 blamed AWSConfluent, Pipedream
AWSown incident not archivedWithin an hour1 vendor, 1 blamed AWSPipedream1 to 4 hours later4 vendors, 4 blamed AWSHubSpot, Xero, Lattice, HeapLater2 vendors, 2 blamed AWSConfluent, Pipedream
Rendered on the server with pele from the Mermaid source below. Each box lists the first few vendors; every vendor is listed further down.
Mermaid source
flowchart LR
  root["AWS<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>1 vendor, 1 blamed AWS<br/>Pipedream"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>4 vendors, 4 blamed AWS<br/>HubSpot, Xero, Lattice, Heap"]:::blames
  root --> h4
  later["Later<br/>2 vendors, 2 blamed AWS<br/>Confluent, Pipedream"]:::blames
  root --> later

Vendors that named AWS

In their own words. “Blames” means a causal phrase sits next to the name (“due to”, “upstream provider”…).

Within an hour

1 vendor

  • +0 min

    Blames AWS

    Major incident

    PipedreamElevated error rates in workflows

    …Elevated error rates in workflows We've implemented a workaround to the AWS issue, and event delivery for workflows should be working again. Delivery of events to Pipedream destinations ($send.http, $…

1 to 4 hours later

4 vendors

Later

2 vendors

Same window, did not name anyone

21 incidents started during the event without naming AWS or another vendor. Some are part of it, some are a normal day’s background (about 19 in these hours).

Show all 21

Detected automatically from 11 status pages opening incidents in the peak hour (normally at most 4). Mentions are matched by name in each incident’s title and updates, then classified as cause, location or integration; quoted text is the vendor’s. Vendors that share one status page are shown once.