Changes / Confluent / Flink job submission and job scale operations are affected in GCP asia-northeast1, europe-west8, europe-north1, southamerica-east1

Confluent: Major incident: Flink job submission and job scale operations are affected in GCP asia-northeast1, europe-west8, europe-north1, southamerica-east1

Summary

Confluent reported a major outage affecting Flink job submission and scaling in GCP asia-northeast1, europe-west8, europe-north1, and southamerica-east1, leaving jobs Pending or Stopping. Symptoms began 2025-12-02 19:53 UTC; the recorded incident started 2025-12-02 21:37 UTC and lasted 4 h 26 min. Fixes restored all affected regions; resolution was recorded at 2025-12-03 02:04 UTC.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 90%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Flink job submission and job scale operations are affected in GCP asia-northeast1, europe-west8, europe-north1, southamerica-east1
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2025-12-02 21:37 UTC
AddedResolved time#5
Before: not present
After: 2025-12-03 02:04 UTC
AddedAffected components#6
Before: not present
After: Confluent Cloud
AddedUpdateinvestigating#7
Before: not present
After: This manifests itself in new Flink jobs remaining stuck in "Statement Status: Pending", existing Flink jobs being requested to stop remaining stuck in "Statement Status: Stopping". This issue started manifesting at 2025-12-02 19:53 UTC. We are currently investigating the issue and attempting to mitigate or provide a work-around.
AddedUpdateidentified#8
Before: not present
After: The root cause for the issue has been identified and the team is working on applying a mitigation.
AddedUpdateidentified#9
Before: not present
After: The team has successfully applied a fix to asia-northeast1 and we're seeing recovery there. Meanwhile the team is applying mitigations in the other affected regions.
AddedUpdatemonitoring#10
Before: not present
After: A fix has been applied to all regions and we've seen recovery in all affected regions. We'll continue to monitor for the next 30 minutes.
AddedUpdateresolved#11
Before: not present
After: No further issues have been observed. The incident is now resolved as of 2025-1203 02:03 UTC.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Flink job submission and job scale operations are affected in GCP asia-northeast1, europe-west8, europe-north1, southamerica-east1 (this change)
Confluent reported a major outage affecting Flink job submission and scaling in GCP asia-northeast1, europe-west8, europe-north1, and southamerica-east1, leaving jobs Pending or Stopping. Symptoms began 2025-12-02 19:53 UTC; the recorded incident started 2025-12-02 21:37 UTC and lasted 4 h 26 min. Fixes restored all affected regions; resolution was recorded at 2025-12-03 02:04 UTC.
Major incident

All Confluent changes → · Confluent profile · Markdown · JSON