Doppler: Secrets unable to be read

Impact: criticalStatus: resolvedStarted 2022-08-29 00:38 UTCResolved 2022-08-29 01:38 UTC (59 min)3 updatesSource: stspg.io

Summary

Doppler reported a critical outage preventing secrets from being read via the API and Dashboard, starting 2022-08-29 00:38 UTC and resolved at 2022-08-29 01:38 UTC (59 min). A Redis server ran out of memory; Doppler increased its allotted memory and said it would add resource-usage alerting.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: API (api.doppler.com)Dashboard (dashboard.doppler.com)

Timeline of updates

investigating2022-08-29 00:38 UTC
We are currently investigating this issue.
monitoring2022-08-29 01:32 UTC
A fix has been implemented and we are monitoring the results.
resolved2022-08-29 01:38 UTC
This issue was caused by a redis server running out of memory. We've increased the amount of memory allotted to this server. We'll also be setting up additional alerting on redis resource usage so that we can catch and prevent these issues in the future.

Rewindex changes for this incident

Critical incident: Secrets unable to be read 2022-08-29 00:38 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Doppler profile · Recent Doppler incidents · Markdown · JSON