Changes / Doppler / Migration Script Race Condition
Doppler: Critical incident: Migration Script Race Condition
Summary
Doppler reported a critical outage starting and resolved at 2020-02-06 23:43 UTC (computed duration: 0 min; vendor reported approximately 2 minutes of downtime). A migration failed to run before deployment, causing services to crash because a critical database table lacked a column; the deployment orchestrator reverted to the previous build.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Migration Script Race Condition
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2020-02-06 23:43 UTC
Before: not present
After: 2020-02-06 23:43 UTC
Before: not present
After: We had a migration script that failed to run before the rest of our services deployed. This caused the newly deployed services to crash because the database was missing a column in a critical table. Our deployment orchestrator quickly detected the issue and reverted to the previous build. Total downtime was approximately 2 minutes.
We plan to mitigate future failures by requiring our migrations to complete before rolling out our remaining services.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Migration Script Race Condition (this change) Doppler reported a critical outage starting and resolved at 2020-02-06 23:43 UTC (computed duration: 0 min; vendor reported approximately 2 minutes of downtime). A migration failed to run before deployment, causing services to crash because a critical database table lacked a column; the deployment orchestrator reverted to the previous build. | Major incident |