Changes / Asana / Asana unreachable for some users

Asana: Major incident: Asana unreachable for some users

Summary

Asana reported a major outage starting 2024-07-01 22:30 UTC that prevented some users from connecting; the recorded duration is 0 min. Its postmortem attributes the outage to an invalid configuration affecting approximately 18% of customers in its US datacenter, with resolution delayed by an unrelated infrastructure configuration management tooling issue. Asana said the issue was fixed.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Asana unreachable for some users
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2024-07-01 22:30 UTC
AddedResolved time#5
Before: not present
After: 2024-07-01 22:30 UTC
AddedUpdateresolved#6
Before: not present
After: Some users were unable to connect to Asana; we investigated and have fixed the issue. We will post a postmortem with additional information.
AddedUpdatepostmortem#7
Before: not present
After: Incident: On 2024-07-01 22:30 UTC, changes related to ongoing work towards smaller, independent deployment environments resulted in an invalid configuration affecting approximately 18% of our customers in our US datacenter. Resolution was delayed due to an unrelated issue with our infrastructure configuration management tools, but the impact was resolved for half of the affected users at 23:35 and for all affected users by 23:42. Impact: Affected customers were unable to open our application \(either with a web browser or with our desktop application\). Existing sessions worked, and no data was lost. Mobile and API were unaffected. Users in other data centers \(and ~82% of users in the US datacenter\) were unaffected by this event. Moving forward: We have done a retrospective analysis using the [5 Whys process](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/), and have identified bugs, process improvements, and monitoring improvements to avoid future downtime while rolling out more fine-grained deployment environments. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Asana unreachable for some users (this change)
Asana reported a major outage starting 2024-07-01 22:30 UTC that prevented some users from connecting; the recorded duration is 0 min. Its postmortem attributes the outage to an invalid configuration affecting approximately 18% of customers in its US datacenter, with resolution delayed by an unrelated infrastructure configuration management tooling issue. Asana said the issue was fixed.
Major incident

All Asana changes → · Asana profile · Markdown · JSON