Rewindex

CompaniesFlagsmithIncidents

Flagsmith: Small number of intermittent requests to Edge API failing

MinorStatus: postmortemStarted Resolved (3 d)9 updatesstspg.io (opens in new tab)

Affected components: Edge API

Timeline of updates

  1. investigating

    We are currently investigating the Edge API sporadically timing out some requests or refusing connections.

  2. investigating

    The issue seems to only be affecting clients connecting to the us-east-2 region. We're currently redirecting traffic away from the region.

  3. monitoring

    We have migrated all traffic away from us-east-2 and are monitoring the impact.

  4. investigating

    We have had further reports of similar issues following the migration of traffic away from us-east-2, we are continuing to investigate.

  5. investigating

    We have passed on all of the troubleshooting information to our infrastructure provider and are awaiting further information from them on this issue. If you are affected by this issue, please get in touch with us at support@flagsmith.com with any information that you can share so we can identify this intermittent issue.

  6. identified

    The issue has been isolated to a specific component in our hosting provider's network infrastructure. We are working with them, and escalating to get the issue resolved.

  7. monitoring

    Our infrastructure provider has confirmed they were intermittently unavailable between December 14, 01:00 UTC and December 17, 22:50 UTC. We are working with affected customers to confirm that this issue is resolved.

  8. resolved

    We have received confirmation from multiple customers in the US east region now that the issue has been resolved.

  9. postmortem

    ## What Happened On Monday 16th December at around 14:25 GMT, we started receiving reports from customers that they were seeing elevated errors in their monitoring regarding connections to the Flagsmith API. Our investigation suggested that there were no application level issues in the Flagsmith platform. Since all customer reports were coming from Eastern US regions, we moved traffic away from the region, redirecting to our US west region. By this time, we were also able to carry out our own testing using infrastructure set up in US east. Our testing, and feedback from our customers showed that moving the traffic away did not resolve the issue. At 19:15 GMT, we opened a ticket with our infrastructure partner. Their investigations also confirmed that there were no issues with the Flagsmith platform itself. At 21:53 GMT, we created a ticket directly with our infrastructure provider \(AWS\) for them to investigate. At around 23:30 GMT on 16th December, we implemented a work around by creating a new DNS record to point directly to our infrastructure in US east, bypassing the latency based routing \(as provided by AWS Global Accelerator\). We shared this with customers experiencing the issues, and they confirmed that it resolved the issue. From here, we continued to investigate the issue with AWS support, providing them with additional information based on our testing, and reports from our customers. On 18th December at 17:58, we received the following information from AWS confirming that there had been an issue with Global Accelerator in the US East region. > The team confirmed that between December 13 5:00 PM PST and December 17 2:50 PM PST, AWS Global Accelerator experienced intermittent connection failures for client traffic served by the Ashburn, Virginia edge location. The issue has been resolved and the service is operating normally. Following this, we were able to confirm that the issue was no longer reproducible for us, or for our customers. ## What’s Next? We have requested, and are currently waiting for, a full post-mortem from the AWS team which may affect our next steps. In the meantime, we have begun looking at alternative solutions for the Global Accelerator that we may be able to keep as a cold standby in case of similar issues in the future.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Flagsmith profileRecent Flagsmith incidentsMarkdownJSON