Companies/FireHydrant/Incidents

FireHydrant: FireHydrant integrations impacted

Affected components: JiraZoomDatadogConfluence CloudAPIEmail notificationsWhatsApp deliveryVoice callsSMS DeliveryAuthenticationSignals

Timeline of updates

1. investigating2025-10-20 14:59 UTC

   Amazon Web Services (AWS) is currently experiencing an outage which is affecting our ability to connect with multiple third-party integrations.

   The FireHydrant application is operational; however, runbook steps associate with multiple integration partners may fail or be delayed. This includes Atlassian (Jira, Confluence, Opsgenie), PagerDuty, Slack, Zendesk, and Zoom. Additionally, there may be delays with alert delivery via SMS and Whatsapp.

   See https://health.aws.amazon.com/health/status (opens in new tab) for more details.

   We will resolve this incident once we have received confirmation the AWS issue has been resolved.

2. identified2025-10-20 17:02 UTC

   We are continuing to monitor the AWS outage and it's affect on our upstream providers and integration partners.

   Impacts to both Twilio and SendGrid are causing delays or failures in the ability to deliver alert notifications via SMS, Whatsapp, and email. Additionally, Slack messaging is affected.

   FireHydrant push notificaiton are unaffected by the outage and are the best option for you to continue receiving alerts.

   You can monitor the availability of our upstream providers as follows:
   - Twilio: https://status.twilio.com/ (opens in new tab)
   - SendGrid: https://status.sendgrid.com/ (opens in new tab)
   - Slack: https://slack-status.com/2025-10/51d84290cf5c8e22 (opens in new tab)

   You can monitor the AWS outage here: https://health.aws.amazon.com/health/status (opens in new tab)

   We will provide updates as they're available.

3. investigating2025-10-20 18:09 UTC

   We are continuing to monitor the AWS outage and it's affect on our upstream providers and integration partners.

   Impact to both Slack and Recall.ai is preventing or delaying the ability of the FireHydrant Scribe to join Zoom and Google Meet calls.

   You can monitor the AWS outage here: https://health.aws.amazon.com/health/status (opens in new tab)

   We will provide updates as they're available.

4. investigating2025-10-20 19:42 UTC

   We are investigating the effect of the AWS outage on our feature flags.

   This impacts users' ability to access features and settings in the FireHydrant UI, but we will provide another update once the full scope of the impact is understood.

5. investigating2025-10-20 19:42 UTC

   We are investigating the effect of the AWS outage on our feature flags.

   This impacts users' ability to access features and settings in the FireHydrant UI, but we will provide another update once the full scope of the impact is understood.

6. identified2025-10-20 20:17 UTC

   Our feature flag provider, LaunchDarkly, is experiencing an outage. This is causing a number of downstream issues, namely an inability to send ad-hoc alerts via /fh page or "Send an Alert" in the UI.

   We are currently putting in a fix to bypass the Signals-related feature flags to restore manual paging.

   Automated alert ingestion is still working so alerts generated from a team's alert rule or existing runbook steps remain operational.

   You can follow LaunchDarkly's outage status here: https://status.launchdarkly.com/ (opens in new tab)

7. identified2025-10-20 20:25 UTC

   Currently testing the fix to restore manual paging via Signals

8. resolved2025-10-20 20:59 UTC

   LaunchDarkly is slowly recovering, but in the meantime we're simultaneously restoring functionality for flag-based features.

   You can continue to monitor their status here: https://status.launchdarkly.com/ (opens in new tab)

9. resolved2025-10-20 21:36 UTC

   We've restored manual paging via Signals. Paging via alert rules and runbooks remained operational throughout this incident. Push notifications continue to be the best way to ensure alert notification delivery given continued service degradation for upstream providers effecting SMS, Whatsapp, and email.

   Additionally, we patched an issue where users with a custom role were unable to access FireHydrant as expected.

   AWS is also reporting that service recovery continues to improve. You can continue to monitor their status here: https://health.aws.amazon.com/health/status (opens in new tab)

10. resolved2025-10-20 22:07 UTC

    LaunchDarkly is reporting significant improvements, and we are seeing our feature flags being successfully evaluated.

    AWS is also reporting that they're recovering.

    We're continuing to monitor the situation, as many of our upstream providers and integration partners are still impacted.

    We'll provide material updates as they occur.

11. resolved2025-10-20 23:11 UTC

    AWS Outage: End-of-Day Update

    AWS has resolved their incident; however, we’re continuing to monitor upstream providers and integration partners that were affected by this outage. This includes, but is not limited to:

    • Atlassian (Jira, Confluence, Opsgenie): https://status.atlassian.com/ (opens in new tab)
    • LaunchDarkly: https://status.launchdarkly.com/ (opens in new tab)
    • PagerDuty: https://status.pagerduty.com/posts/dashboard (opens in new tab)
    • Recall.ai
    • SendGrid: https://status.sendgrid.com/ (opens in new tab)
    • Slack: https://slack-status.com/ (opens in new tab)
    • Twilio: https://status.twilio.com/ (opens in new tab)

    Please report unexpected issues with FireHydrant via our shared Slack channel (preferred) or Support ticket (https://firehydrant.zendesk.com/hc/en-us/requests/new (opens in new tab)). We will be monitoring until all associated incidents have resolved.

12. resolved2025-10-21 13:09 UTC

    We have confirmed all our upstream providers and integration partners have reported a full recovery from yesterday's outage. We will continue to monitor the situation and will post a full incident write-up on our blog later today.

Changes Rewindex caught on this incident

• Oct 20, 2025Major incident: FireHydrant integrations impacted

Impact, status and times are the vendor’s own, as last archived. View on status.firehydrant.com (opens in new tab).

FireHydrant profileRecent FireHydrant incidentsAll outages on October 20, 2025FireHydrant in October 2025MarkdownJSON