Changes / ClickHouse Cloud / CDC ClickPipes stuck in maintenance state
ClickHouse Cloud: Major incident: CDC ClickPipes stuck in maintenance state
Summary
CDC ClickPipes became stuck in maintenance during a weekly upgrade, initially in eu-central-1, with services in other regions also affected. ClickHouse Cloud rated the incident major; it ran from 2026-07-22 11:53 UTC to 2026-07-22 14:36 UTC, lasting 2 h 42 min. Services recovered, but the upgrade rollout remained on hold pending confirmation it could avoid the underlying pod networking failure mode.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: CDC ClickPipes stuck in maintenance state
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-07-22 11:53 UTC
Before: not present
After: 2026-07-22 14:36 UTC
Before: not present
After: As part of our weekly upgrade process, CDC ClickPipes in eu-central-1 became stuck in maintenance state. The team is working on a mitigation.
Before: not present
After: Services in eu-central-1 have recovered. We're in the process of manually recovering ~20 services in other regions.
Before: not present
After: Most services have fully recovered. Continuing to roll out the fix to remaining 4 services (all in eu-central-1, us-east-2).
Before: not present
After: All services have recovered; we will continue monitoring the affected nodes and consolidating our findings. Please reach out to our support team to request access to the RCA, when available.
Before: not present
After: The CDC ClickPipes upgrade rollout remains on hold while we confirm we can safely proceed without triggering the underlying pod networking failure mode that triggered the incident.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: CDC ClickPipes stuck in maintenance state (this change) CDC ClickPipes became stuck in maintenance during a weekly upgrade, initially in eu-central-1, with services in other regions also affected. ClickHouse Cloud rated the incident major; it ran from 2026-07-22 11:53 UTC to 2026-07-22 14:36 UTC, lasting 2 h 42 min. Services recovered, but the upgrade rollout remained on hold pending confirmation it could avoid the underlying pod networking failure mode. | Major incident |
All ClickHouse Cloud changes → · ClickHouse Cloud profile · Markdown · JSON