Companies/Trello/Incidents

Trello: Trello is experiencing performance degradation or is unavailable (resolved)

Timeline of updates

1. resolved2016-06-28 16:00 UTC

   UPDATE 12:32 PM EDT - As of right now, the current incident is considered resolved. Our systems team is continuing to find the root cause. UPDATE 12:13 PM EDT - Any users who experienced a “red box” in the upper right will need to reload their page. The site overall stayed up during this incident. UPDATE 12:10 PM EDT - We believe the issue is resolved and we are continuing to actively monitor the situation. Trello may be slow or unavailable due to socket connections being dropped. We’re actively monitoring the situation to make sure our servers stay up. Because we have multiple web servers, for some users, Trello may appear more or less normal, while for others, they may see a “red box” in the top right corner. Postmortem: At 8:57-9:00am EDT Trello lost approximately one third of our active websocket connections. The root cause of this massive drop is suspected to be with our CDN, but still unconfirmed. Normally when a disconnect occurs, even for a large number of clients simultaneously, the clients are able to reconnect automatically behind the scenes for a seamless user experience. These connection drops happened during one of our peak traffic times. The aggressive reconnect logic in our web client code overwhelmed our servers as all users that had been dropped tried to reconnect. Over the next 26 minutes while we were working to address the issue service continued to degrade until the site was unresponsive and we brought it offline at 9:26am. From around 9:30am-10:00 EDT the Trello service was intermittently available for a small percentage of traffic as we attempted to fully re-enable the site and were hindered as the infrastructure was unable to handle the onslaught of web client reconnects in a timely manner, which caused a snowball effect of increasing reconnect attempts. At roughly 10:00am ET we placed a temporary block on some API routes to ease pressure on the servers and were able to bring the site back by enabling the the site for a gradually increasing percentage of requests. At 10:14am 100% of requests were again being handled successfully and performance returned to normal acceptable levels. In addition to provisioning additional server capacity, our engineers are actively addressing how we handle client connections within the Trello service and how our web clients handle reconnecting after websocket failures to ensure that such a scenario does not bring down the entire service in the future. We’ve been able to handle such massive reconnects without severe service degradation in the past and aim to provide a much higher level of resilience than users experienced this morning.

From Trello’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on trello.status.atlassian.com (opens in new tab).

Trello profileRecent Trello incidentsAll outages on June 28, 2016Trello in June 2016MarkdownJSON