Changes / Asana / Elevated error rates
Asana: Major incident: Elevated error rates
Summary
A major Asana outage left a majority of users unable to access Asana, affecting API and App components in Australia, EU, Japan and US. It ran from 2024-06-17 16:44 UTC to 2024-06-17 19:49 UTC (3 h 4 min). Asana attributed failures to load balancer resource saturation under high traffic; a fix was implemented at 06-17 16:52 UTC.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Elevated error rates
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2024-06-17 16:44 UTC
Before: not present
After: 2024-06-17 19:49 UTC
Before: not present
After: API, App, Australia - API, Australia - App, EU - API, EU - App, Japan - API, Japan - App, US - API, US - App
Before: not present
After: We're currently experiencing some difficulties; as a result, a majority of our users are unable to access Asana. Our Development Team is currently working to resolve this issue as soon as possible. Sincere apologies for the inconvenience caused, please keep an eye on this page for the latest updates.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: Incident: Asana uses several load balancers to route incoming requests to the correct backend service. As our volume of traffic increased over time, these load balancers consumed more resources to handle the additional requests. High traffic pushed computing resource utilization past a critical threshold leading to queueing of slow requests and eventual failures. This incident was made worse by certain error reporting mechanisms sending additional requests through the same load balancing infrastructure.
Impact: Between 15:08 and 16:50 UTC on June 17, 2024, requests to load the Asana application, use of the Asana mobile applications, and calls to our API were delayed or failed at an elevated rate. Users in our European, Australian, and Japanese data centers may have experienced additional crashes while using Asana.
Moving forward: In the short term, we have increased the capacity for our load balancing infrastructure to handle our current traffic as well as anticipated growth, and improved the monitoring to alert us before failure. In the longer term, we have efforts underway to replace this component entirely with routing and load balancing infrastructure that automatically scales in response to increases in traffic.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Elevated error rates (this change) A major Asana outage left a majority of users unable to access Asana, affecting API and App components in Australia, EU, Japan and US. It ran from 2024-06-17 16:44 UTC to 2024-06-17 19:49 UTC (3 h 4 min). Asana attributed failures to load balancer resource saturation under high traffic; a fix was implemented at 06-17 16:52 UTC. | Major incident |