# Doppler change history

Filter: kind=all. Query: `?kind=notable|all|pages|pricing|trust|legal|incident|incident_edit|incident_deleted|incident_update&history=<history_id>&page=N`. Company profile: [Doppler](https://rewindex.com/company/doppler.md)
## Changes by kind

| Kind | Changes | Notable | Latest (UTC) |
| --- | --- | --- | --- |
| Major incident | 25 | 25 | 2025-11-20 15:55 UTC |

## Timeline

| When (UTC) | Kind | Change | Label | Summary |
| --- | --- | --- | --- | --- |
| 2025-11-20 15:55 UTC | Major incident | [Major incident: Background job delays impacting sync, webhook and email delivery](https://rewindex.com/change/i0f2bfb9ec34c28.md) | degradation | Doppler reported a major incident delaying integration sync, webhooks and email delivery, starting at 2025-11-20 15:55 UTC and resolved at 2025-11-20 19:46 UTC; the reported duration was 3 h 50 min. A fix was implemented by 18:54 UTC, but no cause or fix details were stated. |
| 2025-11-18 11:48 UTC | Major incident | [Critical incident: System Outage](https://rewindex.com/change/i1816821758f669.md) | outage | Doppler reported a critical outage affecting Dashboard and API access due to an ongoing Cloudflare outage; direct API requests were unreliable, while integrations accessing secrets were expected to remain unaffected. It started at 2025-11-18 11:48 UTC and was resolved at 2025-11-18 14:59 UTC, lasting 3 h 11 min. |
| 2025-08-20 18:52 UTC | Major incident | [Critical incident: Background job delays impacting sync and email delivery](https://rewindex.com/change/i0e3871723606f4.md) | degradation | Doppler reported a critical background-job degradation from 2025-08-20 18:52 UTC to 2025-08-20 19:58 UTC, lasting 1 h 6 min. Integration syncs showed failed with "Out of sync", while email, activity log, and webhook delivery were delayed. A fix was implemented and all unprocessed jobs were processed by 19:38 UTC; the incident was resolved at 19:58 UTC. |
| 2025-06-12 18:29 UTC | Major incident | [Major incident: System Outage](https://rewindex.com/change/ibf70beec1a9e80.md) | outage | Doppler reported a major system-wide outage affecting API (api.doppler.com) and Dashboard (dashboard.doppler.com), from 2025-06-12 18:29 UTC to 2025-06-12 21:07 UTC, lasting 2 h 38 min. Doppler linked the outage to GCP and Cloudflare disruptions; services were partially restored as those providers recovered, and the incident was resolved. |
| 2025-04-11 02:15 UTC | Major incident | [Critical incident: Extended Maintenance](https://rewindex.com/change/id7cec5b5dbbc1b.md) | outage | Doppler reported critical downtime affecting API (api.doppler.com) and Dashboard (dashboard.doppler.com) due to extended database maintenance. The incident started at 2025-04-11 02:15 UTC and was resolved at 2025-04-11 02:33 UTC, lasting 17 min. |
| 2023-08-04 22:28 UTC | Major incident | [Major incident: Unable to fetch secrets from the Doppler dashboard and API](https://rewindex.com/change/iabb014d5875755.md) | outage | Doppler reported a major outage preventing customers from fetching secrets through the dashboard and API, starting 2023-08-04 22:28 UTC and resolved at 2023-08-04 22:50 UTC (22 min). The vendor attributed the incident to a faulty Kubernetes NetworkPolicy change and said it would evaluate deployment procedures to catch such changes. |
| 2023-01-16 20:37 UTC | Major incident | [Major incident: Unable to sync secrets, webhooks, and activity log notifications](https://rewindex.com/change/i049c2311e00c13.md) | outage | Doppler reported a major partial outage blocking secret syncs, webhooks, activity-log notifications, and API-client secret updates. Records list a start of 2023-01-16 20:37 UTC and resolution at 2023-01-16 19:11 UTC, while the postmortem reports 2023-01-16 19:11 UTC to 2023-01-16 20:04 UTC; duration is unknown. |
| 2023-01-05 00:48 UTC | Major incident | [Major incident: Heroku sync integrations are failing](https://rewindex.com/change/i6e42eceac50959.md) | outage | Doppler’s major Heroku sync outage ran from 2023-01-05 00:48 UTC to 2023-01-05 18:23 UTC (17 h 34 min), blocking new and existing syncs while previously synced secrets remained available. Resolution required all workplaces to reconnect Heroku integrations from workplace Settings, triggering pending changes to sync. |
| 2022-08-29 00:38 UTC | Major incident | [Critical incident: Secrets unable to be read](https://rewindex.com/change/idda682c476950a.md) | outage | Doppler reported a critical outage preventing secrets from being read via the API and Dashboard, starting 2022-08-29 00:38 UTC and resolved at 2022-08-29 01:38 UTC (59 min). A Redis server ran out of memory; Doppler increased its allotted memory and said it would add resource-usage alerting. |
| 2022-06-21 06:33 UTC | Major incident | [Critical incident: Services responding with 500 Internal Server Error](https://rewindex.com/change/i265b5307ae4c13.md) | outage | Doppler’s API (api.doppler.com) and Dashboard (dashboard.doppler.com) returned 500 Internal Server Error responses during a critical outage from 2022-06-21 06:33 UTC to 2022-06-21 07:22 UTC, lasting 49 min. Doppler attributed the issue to a widespread Cloudflare outage and reported implementing a fix before resolution. |
| 2021-12-22 16:50 UTC | Major incident | [Critical incident: Unable to connect to the Doppler dashboard and API](https://rewindex.com/change/i9f8d4e9e59e984.md) | outage | Doppler reported a critical outage preventing connections to its API (api.doppler.com) and Dashboard (dashboard.doppler.com), starting at 2021-12-22 16:50 UTC. A fix was implemented by 2021-12-22 16:54 UTC, and the incident was resolved at 2021-12-22 17:08 UTC; the computed duration was 17 min. |
| 2021-12-08 02:20 UTC | Major incident | [Major incident: Reading/writing secrets intermittently failing due to an outage at our tokenization provider](https://rewindex.com/change/i7ea1608b08c6b7.md) | outage | Doppler reported a major outage causing intermittent failures reading and writing secrets in the API (api.doppler.com) and Dashboard (dashboard.doppler.com). It started at 2021-12-08 02:20 UTC and was resolved at 2021-12-08 10:20 UTC, lasting 8 h 0 min; Doppler attributed it to its tokenization provider being impacted by the AWS us-east-1 outage. |
| 2021-12-07 18:32 UTC | Major incident | [Major incident: Reading/writing secrets failing due to an outage at our tokenization provider](https://rewindex.com/change/i450090697486c6.md) | outage | Reading and writing secrets failed across Doppler’s API and Dashboard in a major-impact outage from 2021-12-07 18:32 UTC to 2021-12-07 19:01 UTC (28 min). Doppler attributed the failure to its tokenization provider being impacted by the AWS us-east-1 outage; all services were operational by 2021-12-07 18:45 UTC. |
| 2021-11-19 23:48 UTC | Major incident | [Major incident: Dashboard intermittently returning bad gateway errors](https://rewindex.com/change/i61e2bbe5d9a6d9.md) | degradation | Doppler’s Dashboard (dashboard.doppler.com) intermittently returned bad gateway errors in a major-impact incident starting 2021-11-19 23:48 UTC, resolved at 2021-11-20 00:58 UTC (1 h 9 min). API and CLI usage were unaffected. Doppler suspected connectivity between Cloudflare and its hosting provider and reported identifying and resolving the issue before monitoring the fix. |
| 2021-11-16 17:39 UTC | Major incident | [Critical incident: All services returning 404](https://rewindex.com/change/ieae8c970277994.md) | outage | Doppler reported a critical outage with all services returning 404, affecting the API (api.doppler.com) and Dashboard (dashboard.doppler.com). It started at 2021-11-16 17:39 UTC and was resolved at 2021-11-16 18:19 UTC, lasting 39 min. Doppler said it appeared related to a larger Google outage; services were coming back online by 18:09 UTC. |
| 2021-11-15 22:14 UTC | Major incident | [Major incident: Users getting rate limited when hitting API](https://rewindex.com/change/i16a7a2e103fbc8.md) | degradation | Doppler reported a major API (api.doppler.com) degradation affecting Service Tokens, which were being rate limited. The incident started at 2021-11-15 22:14 UTC and lasted 27 min, resolving at 2021-11-15 22:41 UTC after Doppler rolled back the code change causing the issue. |
| 2021-10-01 19:34 UTC | Major incident | [Major incident: API unavailable due to contention over database locking](https://rewindex.com/change/i032ced27e94715.md) | outage | Doppler reported a major API outage caused by database lock contention on its Workplace table. Recorded start: 2021-10-01 19:34 UTC; resolution: 2021-10-01 09:00 UTC; computed duration unknown (vendor reported 11 minutes). Service recovered when locks expired; Doppler refactored the affected component to avoid locking. |
| 2021-09-23 17:08 UTC | Major incident | [Major incident: Workplace admins, viewers and collaborators unable to login via CLI](https://rewindex.com/change/ie3fcd88d838d6b.md) | outage | Doppler reported a major API incident preventing workplace admins, viewers and collaborators from logging in via CLI. It started at 2021-09-23 17:08 UTC and was resolved at 2021-09-23 19:12 UTC, lasting 2 h 4 min. Doppler fixed an erroneous permission check that blocked lower privilege users from authorizing the CLI. |
| 2021-04-28 03:15 UTC | Major incident | [Major incident: Secrets storage is unavailable](https://rewindex.com/change/i25b414944ec53b.md) | outage | Doppler reported a major secrets-storage outage affecting API (api.doppler.com) and Dashboard (dashboard.doppler.com), starting 2021-04-28 03:15 UTC and lasting 11 min until resolution at 2021-04-28 03:26 UTC. Doppler said the incident involved a third-party vendor; a fix was implemented and all secrets access was restored. |
| 2021-04-14 04:38 UTC | Major incident | [Major incident: CLI Tokens Authentication Issues](https://rewindex.com/change/iaca899bf3b21c5.md) | outage | Doppler reported a major API (api.doppler.com) incident: CLI token authentication failed with "Invalid CLI Token" errors. It started at 2021-04-14 04:38 UTC and was resolved at 2021-04-14 05:44 UTC, lasting 1 h 5 min. Affected users could work around it by re-logging in with 'doppler login'; Doppler confirmed tokens were functioning again. |
| 2020-07-17 21:20 UTC | Major incident | [Critical incident: DNS resolution failure](https://rewindex.com/change/i0427b2986247d8.md) | outage | Doppler reported a critical DNS outage affecting doppler.com and all subdomains, including API, Dashboard, Deploy, documentation and status-page endpoints, from 2020-07-17 21:20 UTC to 2020-07-17 22:05 UTC (45 min). DNS resolution returned to normal; Doppler initially suspected a Cloudflare outage and reported no data loss. |
| 2020-07-10 03:38 UTC | Major incident | [Critical incident: Downtime due to migration script](https://rewindex.com/change/ia9664ae518f122.md) | outage | Doppler reported a critical outage starting 2020-07-10 03:38 UTC, lasting 28 min and resolved at 2020-07-10 04:07 UTC. All primary and failover production servers went down, disabling dashboard and API access; Deploy was also listed as affected, with no data loss reported. The incident title attributed the downtime to a migration script; a fix was implemented. |
| 2020-02-06 23:43 UTC | Major incident | [Critical incident: Migration Script Race Condition](https://rewindex.com/change/i9f28819ce96ca7.md) | outage | Doppler reported a critical outage starting and resolved at 2020-02-06 23:43 UTC (computed duration: 0 min; vendor reported approximately 2 minutes of downtime). A migration failed to run before deployment, causing services to crash because a critical database table lacked a column; the deployment orchestrator reverted to the previous build. |
| 2019-12-09 05:00 UTC | Major incident | [Critical incident: Restricted API Key authentication is down](https://rewindex.com/change/idaef982d619a5e.md) | outage | Doppler reported a critical Restricted API Key authentication outage affecting API (api.doppler.com) and Deploy (deploy.doppler.com), starting 2019-12-09 05:00 UTC and resolved 2019-12-09 07:38 UTC (2 h 38 min). An authentication-revamp logic bug blocked project access with “User must be provided.” Doppler deployed a fix and reported no data loss. |
| 2019-03-06 23:45 UTC | Major incident | [Critical incident: Users are unable to connect to Doppler servers](https://rewindex.com/change/ie0c1475af41e45.md) | outage | Doppler reported a critical outage preventing connections to its API (api.doppler.com), Dashboard (dashboard.doppler.com), and Website (doppler.com), disrupting customer access. It started at 2019-03-06 23:45 UTC and was resolved at 2019-03-07 00:50 UTC, lasting 1 h 5 min. Doppler implemented fixes and reported no data loss. |


---
Canonical: https://rewindex.com/company/doppler/changes · Markdown: https://rewindex.com/company/doppler/changes.md · JSON: https://rewindex.com/api/v1/companies/doppler/changes · Generated 2026-10-18T02:31:23.251Z