Rewindex

RewindexOutage eventsJul 28, 2020

AWS outage, Jul 28, 2020

18 vendors posted 18 incidents between 08:00 and 11:00 UTC, against about 3 incidents from 3 vendors in the same hours on an ordinary week. 5 vendors named AWS in 5 incidents (4 of them blamed it). Vendors reported an AWS outage; AWS’s own incident for this date is not in the archive.

Observed, not verified. A line means a vendor’s incident named another vendor, or started in the same window. It is not a proven cause.

Times are UTC; lags are from when the first vendor named AWS.

    7. first vendor named AWS 08:30
Named AWSNamed another vendorDid not name anyoneEach column is 30 minutes

How it spread

Vendors that named AWS, grouped by when they posted, and vendors that blamed one of the vendors AWS took down. Solid lines: the vendor named the other. Dashed: same window, not named.

Swipe sideways to see the whole diagram.

AWSown incident not archivedWithin an hour3 vendors, 3 blamed AWSHoneycomb, Twilio, Chargebee1 to 4 hours later2 vendors, 1 blamed AWSStream (GetStream), Elastic Cloud
AWSown incident not archivedWithin an hour3 vendors, 3 blamed AWSHoneycomb, Twilio, Chargebee1 to 4 hours later2 vendors, 1 blamed AWSStream (GetStream), ElasticCloud
Rendered on the server with pele from the Mermaid source below. Each box lists the first few vendors; every vendor is listed further down.
Mermaid source
flowchart LR
  root["AWS<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>3 vendors, 3 blamed AWS<br/>Honeycomb, Twilio, Chargebee"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>2 vendors, 1 blamed AWS<br/>Stream (GetStream), Elastic Cloud"]:::blames
  root --> h4

Vendors that named AWS

In their own words. “Blames” means a causal phrase sits next to the name (“due to”, “upstream provider”…).

Within an hour

3 vendors

  • +0 min

    Blames AWS

    Minor incident

    HoneycombHoneycomb Ingest Outage

    …ours and impacting 0.002% of ingest traffic over that period. The outage was triggered by AWS Route53 suffering lookup failures from EC2 instances in the us-east-1a availability zone (AZ), which hosts…

  • +0 min

    Blames AWS

    Incident, no impact listed

    TwilioFunction invocations failing in US East Coast region due to an AWS DNS outage

    …Function invocations failing in US East Coast region due to an AWS DNS outage Twilio Functions was degraded for 131 minutes between [28 July 2020 1:25 AM PDT and 3:36 AM PDT]. During this period of ti…

  • +30 min

    Blames AWS

    Minor incident

    ChargebeeAWS DNS Resolver Issue

    …AWS DNS Resolver Issue This incident has been resolved. Jobs that had failed due to this incident have been rescheduled. Based on our logs, very few api calls were affected during the incident.…

1 to 4 hours later

2 vendors

  • +2 h 8 min

    Blames AWS

    Minor incident

    Stream (GetStream)Increased API latency

    …Increased API latency AWS Networking issue is now resolved. We are now cleaning up our temporary remediations since they are not needed anymore. Traffic is back to normal for the last hour. AWS Networ…

  • +2 h 20 min

    Names AWS

    Minor incident

    Elastic CloudIncreased DNS resolution latency in us-east-1

    …Increased DNS resolution latency in us-east-1 The underlying issue has been mitigated, and we can see DNS resolution times have been restored to expected levels.…

Same window, did not name anyone

13 incidents started during the event without naming AWS or another vendor. Some are part of it, some are a normal day’s background (about 3 in these hours).

Show all 13

Detected automatically from 8 status pages opening incidents in the peak hour (normally at most 4). Mentions are matched by name in each incident’s title and updates, then classified as cause, location or integration; quoted text is the vendor’s. Vendors that share one status page are shown once.