Flagsmith: Core API: Increased error rates

No impact listedStatus: postmortemStarted 2022-07-01 11:31 UTCResolved 2022-07-01 11:40 UTC (8 min)3 updatesstspg.io

Affected components: Admin APIAPI

Timeline of updates

investigating2022-07-01 11:31 UTC
We are seeing increased 502 responses to our Core API. We are aware of the cause and working on a fix. The Edge API is unaffected.
resolved2022-07-01 11:40 UTC
This incident has been resolved.
postmortem2022-07-01 11:40 UTC
As part of a new feature rollout, there was a large database migration that needed to take place. We knew that the migration would take some time, however, it should not have affected production traffic. Unfortunately, despite our health check returning unhealthy until all migrations are complete, AWS ECS promoted the new version of the API application before the migrations were complete. This meant that the code that was running was expecting certain columns / data to be available in the database which weren’t there yet. We are still investigating what caused ECS to promote the new version before the migrations were complete.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).