ChangesFlagsmithCore API is not responding
Flagsmith: Critical incident: Core API is not responding
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Flagsmith removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Core API is not responding
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2023-09-05 12:01 UTC
Before: not present
After: 2023-09-05 12:01 UTC
Before: not present
After: Admin API, Core API
Before: not present
After: We are currently investigating this issue.
Before: not present
After: We have identified a database migration that has failed as part of a new release. We are working to re-apply the migration.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: This incident has been resolved. A postmortem will follow.
Before: not present
After: **Summary**
On September 5th at 09:45 UTC, we initiated a release that included a database migration aimed at introducing a new constraint to the table containing information related to flags. According to our pre-live tests, this task should not have taken more than 50 milliseconds. Unfortunately, during the release to production, due to the high throughput on a particular table that it needed to acquire a temporary lock on, this caused a backlog of blocked connections waiting on the migration to complete. This caused a knock on effect that exhausted the connections on the database and a full restart was necessary.
Once the restart was complete, the connections were restored and service was resumed. This happened at 10:20 UTC.
**Next Steps**
We have researched the cause of the issue and we do still have further research to understand certain aspects. Our current plan in the meantime is to implement certain safeguards as can be found in the following links to the Postgres documentation which should help reduce any impact in the future.
[https://www.postgresql.org/docs/11/runtime-config-client.html](https://www.postgresql.org/docs/11/runtime-config-client.html)
[https://www.postgresql.org/docs/11/runtime-config-logging.html](https://www.postgresql.org/docs/11/runtime-config-logging.html) \(`log_lock_waits`\)
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Tuesday, September 5, 2023
1 incident change
- CriticalCritical incident
Core API is not respondingthis change