Harness: Prod2 Pipelines with verify step failing intermittently
Affected components: Continuous Delivery - Next Generation (CDNG)Prod 2 - Continuous Delivery - Next Generation (CDNG)
Timeline of updates
- monitoring
A fix has been implemented and we are monitoring the results.
- resolved
This incident has been resolved.
- postmortem
### Summary Customers on Prod 2 experienced intermittent failures of a few Verify steps at startup. Affected steps failed shortly after launch with a message indicating Continuous Verification\(CV\) could not reach an internal Harness service. The issue stemmed from a recent service update that introduced connection pooling to improve performance and reliability with a keep-alive setting incompatible with the receiving service's timeout. No customer data was lost or corrupted. ### Customer Impact Customers using CV Verify steps on Prod-2 saw intermittent step failures during the incident window. The failure rate was low — the majority of executions completed successfully throughout the window. Pipelines that did not use CV Verify steps were unaffected. ### Root Cause A recent CV service update introduced connection pooling to an internal platform component, with a keep-alive duration longer than that component's idle timeout. Connections idle long enough were silently closed by the server; the CV service did not detect this and had no retry logic for such failures, causing them to surface directly as Verify step failures. The issue was intermittent because it only affected connections that happened to be idle long enough to expire. ### Mitigation The connection keep-alive duration was reduced to fall within the receiving service's idle timeout. Failures dropped to near zero immediately, and no further step failures of this type have been observed on any production environment. ### Next Steps To prevent recurrence, Harness will: * **Fix connection handling:** Tuned the default keep-alive and enabled automatic retry on connection failures so stale-connection errors are handled transparently. * **Add resilience at step startup:** Introduce retry with backoff for the internal service call at the Verify step start, so isolated transient failures do not surface as immediate step failures. * **Improve alerting:** Enhanced monitoring for internal connection error rates.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).