Companies / Doppler / incidents
Doppler: Migration Script Race Condition
Doppler reported a critical outage starting and resolved at 2020-02-06 23:43 UTC (computed duration: 0 min; vendor reported approximately 2 minutes of downtime). A migration failed to run before deployment, causing services to crash because a critical database table lacked a column; the deployment orchestrator reverted to the previous build.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Timeline of updates
- resolved
We had a migration script that failed to run before the rest of our services deployed. This caused the newly deployed services to crash because the database was missing a column in a critical table. Our deployment orchestrator quickly detected the issue and reverted to the previous build. Total downtime was approximately 2 minutes. We plan to mitigate future failures by requiring our migrations to complete before rolling out our remaining services.
Rewindex changes for this incident
- Critical incident: Migration Script Race Condition 2020-02-06 23:43 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Doppler profile · Recent Doppler incidents · Markdown · JSON