# Doppler: Major incident: Heroku sync integrations are failing

**Summary:** Doppler’s major Heroku sync outage ran from 2023-01-05 00:48 UTC to 2023-01-05 18:23 UTC (17 h 34 min), blocking new and existing syncs while previously synced secrets remained available. Resolution required all workplaces to reconnect Heroku integrations from workplace Settings, triggering pending changes to sync. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2023-01-05 00:48 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/k0fczv22864d](https://stspg.io/k0fczv22864d)
- [Company: Doppler](https://rewindex.com/company/doppler.md)
- Versions: before (none: first archived) → after incident\_version:308/0f87chchysv8@99aa1098295b
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Heroku sync integrations are failing
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2023-01-05 00:48 UTC
```

**5. Added: Resolved time**

```diff
+ 2023-01-05 18:23 UTC
```

**6. Added: Affected components**

```diff
+ Heroku, Integrations
```

**7. Added: Update** (identified) · 2023-01-05 00:48 UTC

```diff
+ New and existing Heroku sync integrations are currently not functioning. We have identified the issue and are working on resolving it. Secrets previously synced to Heroku apps will remain available but new secret changes in Doppler will not be synced until this is resolved.
```

**8. Added: Update** (identified) · 2023-01-05 01:06 UTC

```diff
+ This incident may persist for up to 24 hours. In the meantime, if you have an urgent secret change that needs to be made, please update the secret in your Doppler dashboard and then manually update the secret in Heroku directly via ʹheroku config:setʹ. Updating the secret in Doppler will ensure that the value in Heroku is not overwritten once the incident is resolved.
```

**9. Added: Update** (resolved) · 2023-01-05 18:23 UTC

```diff
+ This issue is now resolved. Additional action is required to re-enable existing Heroku syncs. All workplaces will need to reconnect their Heroku sync integrations from the [workplace Settings page](https://dashboard.doppler.com/workplace/settings). Once integrations are reconnected, all syncs will automatically be triggered to sync any pending changes in Doppler. A postmortem will be available soon with more details regarding what happened.
```

**10. Added: Update** (postmortem) · 2023-01-05 21:30 UTC

```diff
+ **Root Cause** Doppler syncs secrets to Heroku via a Heroku OAuth application. This application is created in the Heroku dashboard and must be owned by a single Heroku user account. Doppler’s previous Heroku OAuth application was owned by a specific Doppler employee’s Heroku account without access to any additional resources. During a routine external account audit, this account was mistakenly identified as unused and manually deleted by our security team. This irrecoverably deleted Doppler’s existing Heroku OAuth application, thereby breaking any existing syncs and requiring the creation of a new OAuth application in a new account. **Resolution** Because users had authorized our previous Heroku OAuth application to their Heroku account\(s\), users need to authorize the new Heroku OAuth application. This involves reconnecting the integration from the [Doppler dashboard](https://dashboard.doppler.com/workplace/settings). Once the integration is reconnected, Doppler will re-enable all associated syncs that have been disabled and perform a fresh sync. Note that the previous OAuth application was deleted and therefore no action is required to remove its access to your Heroku account. **Next Steps** Internally, we’re reorganizing how shared accounts used for critical functionality are stored in 1Password. This new 1Password organization should help prevent this kind of accidental deletion in the future. We avoid shared accounts whenever possible, but this isn’t always feasible given third party implementations. We'll also be adding our individual integrations to our status page. This will allow customers to more easily see which integrations, if any, are currently experiencing issues.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2023-01-05 00:48 UTC | Major incident: Heroku sync integrations are failing (this change) | outage |

All Doppler changes: [timeline](https://rewindex.com/company/doppler/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i6e42eceac50959 · Markdown: https://rewindex.com/change/i6e42eceac50959.md · JSON: https://rewindex.com/api/v1/changes/i6e42eceac50959 · Generated 2026-10-18T01:29:48.862Z