Before: not present
After: Root Cause Analysis Report
1. Summary
On October 15, 2026, between 12:24 AM and 1:16 AM PT, Prod3 experienced a delay in how quickly execution graphs, aggregations, and observability data updated. The underlying workflow execution engine continued to function normally throughout and note that, this was a data-freshness and UI-responsiveness issue, not a workflow execution failure. The root cause was a backend search-service dependency in our event-processing pipeline that became a bottleneck under an unusually high volume of API calls to a specific execution-details endpoint, concentrated from one customer account. We mitigated by scaling the search cluster, and after a brief recurrence later that morning, doubled its capacity. We subsequently shipped a permanent fix that removed the search-service dependency from this code path entirely, cutting search load by about 95%.
2. Impact
Prod3 users may have seen delayed or stale execution graphs, aggregation views, and observability data for about 52 minutes, with a brief recurrence later the same morning.
Underlying workflow execution was not affected and pipelines continued to run and complete normally throughout.
No confirmed workflow execution failures occurred, and no data was lost.
We temporarily applied protective rate-limiting safeguards on the affected API and coordinated with the customer whose traffic drove the load while a permanent fix was developed and deployed.
3. Root Cause
Our platform processes workflow execution events through an internal, queue-based pipeline. A step in that pipeline performed a synchronous search-service lookup as part of completing each event which is aa dependency that, under normal conditions, added negligible delay.
An unusually high and sustained volume of calls to an execution-details API, concentrated from one customer account, drove a large increase in search-service query volume. This pushed CPU on the search cluster's warm-tier nodes to saturation, which in turn slowed the synchronous lookups in the event-processing pipeline, causing events to back up and execution-graph and observability data to fall behind in near-real time, even though underlying workflow execution did not depend on this same path.
4. Resolution
Verified core databases and recent deployments were not responsible, narrowing the investigation to the search-service dependency.
Identified and mitigated warm-tier search-cluster CPU saturation by scaling cluster capacity during the incident.
Further increased search-cluster capacity after a brief recurrence later the same morning.
Added temporary logging to better identify which traffic was driving load, and worked with the customer to understand their usage pattern.
Applied temporary protective rate-limiting on the affected endpoint as a safeguard.
Shipped a permanent API optimization that removes the search-service dependency from this workflow-event-processing path, which reduced search load by about 95%.
5. Preventive Actions
Add duration-based alerting for sustained search-cluster CPU saturation, correlated with event-processing lag, so prolonged stress escalates automatically rather than depending on manual observation.
Validate search-cluster capacity headroom going forward, and evaluate whether capacity can now be safely reduced given the dependency has been removed from the hot path.
Decouple or bound similar dependencies: audit other workflow-event-processing paths for synchronous search-service calls and inventory similar dependencies elsewhere in the platform.
Add finer-grained rate limiting on the affected endpoint so that one customer's traffic pattern cannot affect other customers sharing the same infrastructure.
Define a clearer policy for applying protective traffic controls when a single customer's usage threatens shared-platform stability.
We appreciate your patience while we worked through a complete resolution, and we remain committed to strengthening the resilience of our platform as a result of this incident.