# Ably: Transient netsplit in us-east-2 leading to protracted netsplit for a few channels

> Ably status incident “Transient netsplit in us-east-2 leading to protracted netsplit for a few channels” on 2024-08-12: impact none (vendor’s label), 0 min, 1 update(s). Archived by Rewindex.

Canonical: https://rewindex.com/company/ably/incidents/lfbb2yvvtghx
Last updated: 2024-08-12T14:08:15Z

- Impact (vendor’s label): none
- Status: resolved
- Started: 2024-08-12 14:08 UTC · Resolved: 2024-08-12 14:08 UTC · Duration: 0 min
- Updates: 1 · Affected components: Ably Pub/Sub Channels - US East (Ohio), US East (Ohio)
- [Vendor status page: https://stspg.io/ch1qm77313qt](https://stspg.io/ch1qm77313qt)
- [Company: Ably](https://rewindex.com/company/ably.md)

## Timeline of updates

- **resolved** 2024-08-12 14:08 UTC: At 03:32 UTC on the 12th, the us-east-2 (Ohio) data center was partially partitioned (netsplit) from the rest of the cluster. The netsplit was healed by 03:40. However, a very small proportion of channels did not self-heal, and remained partitioned between us-east-2 and at least one other site in the cluster (mostly either eu-central-1 or eu-west-1) until we took manual corrective action to fix them, which was completed in the production cluster at 12:22 UTC. Our existing quality-of-service alerting infrastructure should have lead to us being paged at the time, however, due to a bug in the definition for this particular alert, this did not happen. Reliable delivery of messages is the thing we prioritise above all else at Ably, and so even though the number of affected channels was small, we are taking this extremely seriously, and (other than fixing the bug in the self-healing function) we will be prioritising improvements to our alerting framework to ensure that no future similar issue can be allowed to last this long. We are deeply sorry to those customers who had affected channels.

## Cite this

Status incident (vendor impact label: none) posted by Ably, started 2024-08-12 14:08 UTC, resolved after 0 min, 1 update(s).

- Canonical URL: https://rewindex.com/company/ably/incidents/lfbb2yvvtghx
- Citation: Ably, “Transient netsplit in us-east-2 leading to protracted netsplit for a few channels” (2024-08-12 14:08 UTC). Archived by Rewindex: https://rewindex.com/company/ably/incidents/lfbb2yvvtghx (CC BY 4.0).
- Markdown link: `[Ably: Transient netsplit in us-east-2 leading to protracted netsplit for a few channels (2024-08-12)](https://rewindex.com/company/ably/incidents/lfbb2yvvtghx)`


---
Canonical: https://rewindex.com/company/ably/incidents/lfbb2yvvtghx · Markdown: https://rewindex.com/company/ably/incidents/lfbb2yvvtghx.md · JSON: https://rewindex.com/api/v1/companies/ably/incidents/lfbb2yvvtghx · Data licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) · Generated 2026-11-04T16:54:04.961Z