Rewindex

CompaniesCourierIncidents

Courier: Courier Send and Message Event Slowdown

MinorStatus: resolvedStarted Resolved (36 min)3 updatesstspg.io (opens in new tab)
Part of an outage event

AWS outage, Feb 13, 2025 · 11 vendors, 11 incidents

Courier’s incident blames AWS as the cause. (hop 1, posted +0 min vs the first vendor naming AWS)

“…essage Event Slowdown The team has identified the issue related to network connections on AWS which resulted in a backup of messages on our pipeline. Messages are flowing through as expected, and the…”

Swipe sideways to see the whole diagram.

AWSown incident not archivedWithin an hour2 vendors, 1 blamed AWSCourier, Teleport1 to 4 hours later5 vendors, 4 blamed AWSCoralogix, SingleStore, Aptible, Snowflake +1Later1 vendor, 1 blamed AWSGrafana CloudCourier (this incident)posted +0 min
AWSown incident not archivedWithin an hour2 vendors, 1 blamed AWSCourier, Teleport1 to 4 hours later5 vendors, 4 blamed AWSCoralogix, SingleStore, Aptible,Snowflake +1Later1 vendor, 1 blamed AWSGrafana CloudCourier (this incident)posted +0 min
Courier is outlined in red. Groups show up to a few vendors each; the event page lists every incident.
Mermaid source
flowchart LR
  root["AWS<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>2 vendors, 1 blamed AWS<br/>Courier, Teleport"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>5 vendors, 4 blamed AWS<br/>Coralogix, SingleStore, Aptible, Snowflake +1"]:::blames
  root --> h4
  later["Later<br/>1 vendor, 1 blamed AWS<br/>Grafana Cloud"]:::blames
  root --> later
  me["Courier (this incident)<br/>posted +0 min"]:::me
  h1 --- me

Full event: timeline, evidence and all vendors →

Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.

Affected components: API

Timeline of updates

  1. identified

    The Courier team has identified a slowdown in sending notifications. This issue has been identified by our team and is closely monitoring. We do not expect any messages to drop. We will continue to monitor and update accordingly.

  2. monitoring

    Our team has identified a bottleneck in our system caused by timeouts on our sendworker. Our exponential backoff has caught up and messages are starting to go through.

  3. resolved

    The team has identified the issue related to network connections on AWS which resulted in a backup of messages on our pipeline. Messages are flowing through as expected, and the stream has caught up.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Courier profileRecent Courier incidentsMarkdownJSON