Companies / Asana / incidents

Asana: Application and API errors

Affected components: AppJapan - AppMobileUS - MobileAppUS - AppJapan - Webhooks and Event StreamsWebhooks and Event StreamsAutomations and Background ActionsEU - Automations and Background ActionsAutomations and Background ActionsJapan - Automations and Background ActionsAutomations and Background ActionsUS - Automations and Background ActionsUS - Webhooks and Event StreamsWebhooks and Event StreamsAustralia - Webhooks and Event StreamsWebhooks and Event StreamsAPIAustralia - APIAppEU - AppAustralia - MobileMobileAPIJapan - APIAppAustralia - AppJapan - MobileMobileAPIEU - APIAustralia - Automations and Background ActionsAutomations and Background ActionsEU - Webhooks and Event StreamsWebhooks and Event StreamsEU - MobileMobileAPIUS - API

Timeline of updates

1. identified2024-08-01 15:21 UTC

   We are investigating application issues related to overload, and will provide updates.

2. identified2024-08-01 16:04 UTC

   We are continuing to work on a fix for this issue.

3. identified2024-08-01 16:34 UTC

   We are continuing to work on a fix for this issue.

4. identified2024-08-01 16:37 UTC

   We are continuing to work on a fix for this issue.

5. identified2024-08-01 16:58 UTC

   We are continuing to work on a fix for this issue.

6. identified2024-08-01 17:03 UTC

   We are continuing to work on a fix for this issue.

7. identified2024-08-01 17:05 UTC

   We are continuing to work on a fix for this issue.

8. identified2024-08-01 17:33 UTC

   We are continuing to work on a fix for this issue.

9. identified2024-08-01 18:13 UTC

   We are continuing to work on a fix for this issue.

10. monitoring2024-08-01 18:35 UTC

    We've identified the root cause and are monitoring our systems

11. resolved2024-08-01 19:49 UTC

    This incident has been resolved.

12. postmortem2024-08-02 00:16 UTC

    **Incident:** On 2024-08-01 at 14:09 UTC, an increased connection rate on a central database saturated the database’s ability to accept new connections. As a result, Asana became unavailable to users in all regions. In response, Asana engineers throttled traffic for automations, webhooks, and API, and were able to restore primary functionality of the Asana application at 15:01 UTC. The application continued to run with sporadic unavailability until 18:22 UTC, when the underlying cause of the increased connections was determined and disabled. At this point the application was fully operational, but asynchronous workloads such as automations and notifications were delayed due to the work enqueued while traffic was throttled. The incident was resolved when asynchronous work queues fully caught up at 19:39 UTC. **Impact:** From 14:09 UTC to 15:01 UTC, users were unable to access Asana through the web, mobile, or API. From 15:01 UTC to 18:22 UTC users could access Asana through the web and mobile, but experienced sporadic unavailability. Access to the API was restored gradually between 15:15 UTC and 15:58 UTC. From 15:01 UTC to 19:39 UTC users experienced delayed automations, background actions, webhooks, and event streams. No customer data was lost. **Moving forward:** We have reverted the change that caused the increased connections. We are adding monitoring to detect changes in database connection rates, so we can identify potential problems earlier. Beyond this, we will be replicating data from the central database to other databases to reduce the impact of any single database being overloaded.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Asana profile · Recent Asana incidents · Markdown · JSON