Rewindex

CompaniesAptibleIncidents

Aptible: Operations blocked - Route 53 propagation delays

MajorStatus: resolvedStarted Resolved (57 min)3 updatesstspg.io (opens in new tab)
Part of an outage event

AWS outage, Feb 13, 2025 · 11 vendors, 11 incidents

Aptible’s incident blames AWS as the cause. (hop 1, posted +1 h 18 min vs the first vendor naming AWS)

“…Operations blocked - Route 53 propagation delays This incident has been resolved. This incident has been resolved. We are noticing Route 53 record requests succeeding in a normal time frame, and are l…”

Swipe sideways to see the whole diagram.

AWSown incident not archivedWithin an hour2 vendors, 1 blamed AWSCourier, Teleport1 to 4 hours later5 vendors, 4 blamed AWSCoralogix, SingleStore, Aptible, Snowflake +1Later1 vendor, 1 blamed AWSGrafana CloudAptible (this incident)posted +1 h 18 min
AWSown incident not archivedWithin an hour2 vendors, 1 blamed AWSCourier, Teleport1 to 4 hours later5 vendors, 4 blamed AWSCoralogix, SingleStore, Aptible,Snowflake +1Later1 vendor, 1 blamed AWSGrafana CloudAptible (this incident)posted +1 h 18 min
Aptible is outlined in red. Groups show up to a few vendors each; the event page lists every incident.
Mermaid source
flowchart LR
  root["AWS<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>2 vendors, 1 blamed AWS<br/>Courier, Teleport"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>5 vendors, 4 blamed AWS<br/>Coralogix, SingleStore, Aptible, Snowflake +1"]:::blames
  root --> h4
  later["Later<br/>1 vendor, 1 blamed AWS<br/>Grafana Cloud"]:::blames
  root --> later
  me["Aptible (this incident)<br/>posted +1 h 18 min"]:::me
  h4 --- me

Full event: timeline, evidence and all vendors →

Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.

Affected components: Aptible Deploy

Timeline of updates

  1. identified

    We've noticed that some Operations are failing due to Route53 record changes not propagating within the 10 minute time limit allowed by our platform. In order to prevent Apps and Databases DNS records from reaching an inconsistent state, we are temporarily blocking Operations. Performance and reachability of existing Apps and Database is not impacted.

  2. monitoring

    We are noticing Route 53 record requests succeeding in a normal time frame, and are lifting the operation block at this time. We'll continue to observe running operations to ensure stability.

  3. resolved

    This incident has been resolved.

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Aptible profileRecent Aptible incidentsMarkdownJSON