Rewindex

CompaniesCourierIncidents

Courier: Delay in Message Processing

MajorStatus: resolvedStarted Resolved (3.0 h)6 updatesstspg.io (opens in new tab)

Affected components: APICourier - APICourier - ObservabilityObservabilityCourier - Web ApplicationWeb Application

Timeline of updates

  1. identified

    The Courier team identified an issue in our health monitoring involving our message event processing. The issue has been identified and a revert is in place.

  2. monitoring

    Our team has released a revert to address the regression and it's in the process of merging.

  3. monitoring

    The release is live, and the team is monitoring it.

  4. monitoring

    Release is published and building to production. ETA ~45 minutes.

  5. monitoring

    Fix has been deployed, and enqueued messages have started to go through slowly. Once the bottleneck clears, messages should start to flow normally.

  6. resolved

    The general pipeline has recovered.

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Courier profileRecent Courier incidentsMarkdownJSON