Companies / Amplitude / incidents
Amplitude: Funnel charts in error state
Timeline of updates
- resolved
What happened: - On Friday, June 9, 2023 at 6:40pm PST, code with a critical bug in Funnel charts was pushed to production. Any users that visited a Funnels chart from June 9, 2023 at 6:40pm PST - June 11, 2023 at 7:10am PST saw the “Something went wrong” error on their funnel charts. What was the resolution: - On Sunday, June 11, 2023 at 7:05am PST, our on-call engineering team discovered the bug with funnel charts and rolled back the change within 5 minutes, at 7:10am PST. This fixed the issue for all customers at that point in time. - A subsequent fix was created and tested and finally deployed 1h30m later to production with all the changes added in the release on Friday. - Since this put the customer’s UI in a bad state, we ask all customers to refresh Amplitude to restore a clean state for all charts. What is amplitude going to do in the future to prevent this from happening again? - Short-term: Set up additional monitoring to alert on-call teams of extended front-end sentry errors for funnels and any other uncovered charts - Medium-term: Address the root cause of the issue relating to some typing gaps in our current codebase. - Medium/Long-term: Additional e2e test coverage to catch this specific regression + more broad coverage to catch similar regressions in the future.
From Amplitude’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.
Impact, status and times are the vendor’s own, as last archived. View on status.amplitude.com (opens in new tab). Amplitude profile · Recent Amplitude incidents · Markdown · JSON