Rewindex

CompaniesLoopsIncidents

Loops: Primary database outage

MajorStatus: postmortemStarted Resolved (39 min)6 updatesstspg.io (opens in new tab)

Affected components: APIEmail Sending - LoopsLoopsAppSMTP RelayCampaignsEmail Sending - CampaignsEmail Sending - TransactionalTransactional

Timeline of updates

  1. investigating

    We are currently investigating this issue.

  2. monitoring

    A fix has been implemented and we are monitoring the results.

  3. monitoring

    We are continuing to monitor for any further issues.

  4. monitoring

    We are continuing to monitor for any further issues.

  5. resolved

    This incident has been resolved.

  6. postmortem

    A misconfiguration by our database vendor prevented our database from restarting successfully. The misconfiguration was identified and remedied. Campaigns scheduled to send during the outage are now sent. Loop emails scheduled to send during the outage are now sent. Transactional emails or API calls made during the outage window need to be retried.

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Loops profileRecent Loops incidentsMarkdownJSON