Rewindex

CompaniesHarnessIncidents

Harness: FME - Some customers are experiencing delays in scheduled exports of impressions

MinorStatus: postmortemStarted Resolved (24 h)5 updatesstspg.io (opens in new tab)

Affected components: FMEProd 1 - FMEFMEProd 2 - FMEFMEProd 3 - FME

Timeline of updates

  1. monitoring

    A fix has been implemented and we are monitoring the results.

  2. monitoring

    We are seeing the backlogged items worked through and continuing to monitor results.

  3. monitoring

    We are continuing to monitor results while the backlog is worked through.

  4. resolved

    This incident has been resolved.

  5. postmortem

    **Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. ‌ **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. ‌ **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Harness profileRecent Harness incidentsMarkdownJSON