Companies / Doppler / incidents

Doppler: Restricted API Key authentication is down

Affected components: Deploy (deploy.doppler.com)API (api.doppler.com)

Timeline of updates

1. identified2019-12-09 05:00 UTC

   The issue has been identified and a fix is being implemented.

2. monitoring2019-12-09 06:48 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved2019-12-09 07:38 UTC

   This issue has been resolved. During a revamp of our authentication methods, we introduced a logic bug. This bug prevented restricted API Keys from gaining access to their projects, causing any restricted API Key to receive the error "User must be provided." A fix has been deployed and this issue is now resolved.

4. postmortem2019-12-09 09:23 UTC

   A critical outage around Restricted API Key authentication occurred Sunday evening at exactly 9:00 PM PST leading to over 1 hour of downtime. This outage did not result in any data loss but did render Restricted API Keys inoperable with our API endpoints for that period of time. This was an unacceptable amount of downtime and could have been completely prevented. We take your trust very seriously as Doppler is a critical path in your devops and productivity workflows. We have learned from this experience while fixing the root cause and adding checks to prevent this kind of outage in the future. Here is what happened: # Timeline **December 9th, 2019 - 9:00 PM \(PST\)** Our engineering team rolls out new authentication logic for our API endpoints. The update is designed to increase our defense in depth by adding additional layers of authentication around every layer in our core stack. **December 9th, 2019 - 9:40 PM \(PST\)** While testing the API endpoints to verify the new logic, we find that all requests utilizing a Restricted API Key are being rejected. **December 9th, 2019 - 10:00 PM \(PST\)** We identify the culprit bug in our codebase and start working on a patch. The bug is linked to how we handle Restricted API Keys. Doppler’s API offers 3 methods of authentication: Personal Keys, Restricted API Keys \(now called Service Tokens\), and CLI Tokens. Personal Keys and CLI Tokens are tied to a user identity, while Service Tokens are not. Our investigation finds that our authentication logic was requiring a user identity and did not gracefully handle a case where one would not be present. **December 9th, 2019 - 10:48 PM \(PST\)** The patch is released to production and the engineering team starts monitoring the fix. **December 9th, 2019 - 11:38 PM \(PST\)** After stress testing the patched authentication logic in production, we mark the incident resolved. # Moving Forward **End to End Testing \(e2e\)** From unit tests to e2e testing, we strive to test _every_ part of our stack. As it turns out, we had not added e2e tests for Service Tokens. This outage has led us to reassess our test coverage and focus on testing all remaining user flows. **StatusPage Accuracy** Doppler uses Pingdom to test for uptime and display those results on our status page. Pingdom was testing our health check endpoints which do not require authentication. This led Pingdom to not report our outage. We have now changed Pingdom to test our secrets endpoint, which requires multiple layers of authentication and is a better indicator of our API being serviceable to customers. **Announcing Maintenance Windows** Starting today, we are going to announce major maintenance schedules one week in advance. Our goal with maintenance is to _never_ have any downtime, but when downtime does strike, we want you to be prepared. Knowing a time window in advance can help your engineering teams navigate and prepare for the off chance appropriately.

Rewindex changes for this incident

• Critical incident: Restricted API Key authentication is down 2019-12-09 05:00 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Doppler profile · Recent Doppler incidents · Markdown · JSON