Changes / ClickHouse Cloud / ClickPipes failing on Kinesis in AWS us-east-1
ClickHouse Cloud: Critical incident: ClickPipes failing on Kinesis in AWS us-east-1
Summary
ClickHouse Cloud reported critical ClickPipes failures on Kinesis in AWS us-east-1, starting 2026-09-15 18:57 UTC and resolved 2026-09-17 00:43 UTC (29 h 45 min). AWS identified a Kinesis IAM authorization defect and began rolling back the change; ClickHouse confirmed Enhanced Fanhout (EFO) Kinesis pipes were fully operational at resolution.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: ClickPipes failing on Kinesis in AWS us-east-1
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2026-09-15 18:57 UTC
Before: not present
After: 2026-09-17 00:43 UTC
Before: not present
After: Currently investigating failures on AWS us-east-1 for customers using ClickPipes on Kinesis in this region.
Before: not present
After: The issue has been identified as a change on AWS related to Kinesis IAM permissions and appears to be currently isolated to us-east-1. While we work with AWS to mitigate this issue, we recommend to update your IAM policy to include a trailing wildcard (/*) to the resource for the kinesis:RegisterStreamConsumer action like so:
{
"Effect": "Allow",
"Action": [
"kinesis:RegisterStreamConsumer"
],
"Resource": [
"arn:aws:kinesis:{REGION}:{ACCOUNT_ID}:stream/{STREAM_NAME}/*"
]
}
Once this is applied, you must restart your ClickPipe for the changes to take affect.
Before: not present
After: AWS has confirmed the issue on their end and are in the process of rolling back the change. The ETA for this rollback in us-east-1 is by end of day tomorrow. Signs of gradual recovery are expected in us-east-1 as the rollback progresses across AZs during this time. In the meantime, if your ClickPipe is still affected, please update your IAM policy as instructed in the previous status update.
Before: not present
After: AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>/consumer/<consumer-name>) rather than the stream resource (for example, arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException.
AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.
Before: not present
After: AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>/consumer/<consumer-name>) rather than the stream resource (for example, arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException.
AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.
Before: not present
After: This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues: https://console.clickhouse.cloud/support
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: ClickPipes failing on Kinesis in AWS us-east-1 (this change) ClickHouse Cloud reported critical ClickPipes failures on Kinesis in AWS us-east-1, starting 2026-09-15 18:57 UTC and resolved 2026-09-17 00:43 UTC (29 h 45 min). AWS identified a Kinesis IAM authorization defect and began rolling back the change; ClickHouse confirmed Enhanced Fanhout (EFO) Kinesis pipes were fully operational at resolution. | Major incident |
All ClickHouse Cloud changes → · ClickHouse Cloud profile · Markdown · JSON