Companies / Smartsheet / incidents
Smartsheet: Smartsheet Application & API
Timeline of updates
- resolved
The Smartsheet application is healthy. On February 2, at 8:30 AM PST, a back end message queueing system experienced a failure that held connections open to the database as well as other systems. This stopped processing for all requests to the Smartsheet application causing an unplanned system outage. Smartsheet’s operations and development teams immediately engaged to troubleshoot and remedy the situation. For the next two hours our engineers worked to determine root cause and return functionality to the system. Initially noticing the flood of open connections, a series of partial and full restarts of the front end application servers was attempted with no impact to functionality. Back end batch services were also halted to ease pressure on the system. During this time thread dumps were taken which lead to the discovery of a problem with the message queueing system. At 10:20 AM PST our message queueing system was turned off in order to perform a rebuild of the queueing system. This allowed the core application to be available at 10:30 AM but updates to systems such as Search, Reporting, Resource Management, Webhooks, Connectors, and Email were delayed. The message queue was brought back online at 11:55 AM PST allowing new requests to these systems and restoring system availability. Based on lessons learned from yesterday’s outage, Smartsheet will: Add more monitoring on all core systems. Further refine our operating and communications playbooks for addressing such issues. Increase investments in reliability of components. Add more redundancy to our network.
From Smartsheet’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.
Impact, status and times are the vendor’s own, as last archived. View on status.smartsheet.com (opens in new tab). Smartsheet profile · Recent Smartsheet incidents · Markdown · JSON