rewindex

CompaniesGitHubIncidents

GitHub: Delays for Pull Requests and Webhook deliveries

CriticalStatus: investigatingeditedStarted Not resolved12 updatesstspg.io (opens in new tab)

Affected components: WebhooksActionsPull Requests

Discussed on Hacker News

Threads are matched automatically by vendor name and timing; points and comments refresh every 30 minutes. Method and limits.

Timeline of updates

  1. investigating

    We are investigating reports of degraded performance for Actions, Pull Requests and Webhooks

  2. investigating

    Actions is experiencing degraded performance. We are continuing to investigate.

  3. investigating

    We are investigating delays affecting pull request updates and webhook deliveries. Increased load on our event streaming platform is contributing to hotspots. We are shifting traffic away from affected areas and adding processing capacity; impact is ongoing while these mitigations take effect.

  4. investigating

    We continue to see delays affecting pull request updates, webhook deliveries, and audit log events. The event streaming platform is showing signs of recovery, and we are monitoring as the impact continues.

  5. investigating

    Audit log events and Actions workflow status updates continue to be delayed. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring the recovery and will share further updates as we learn more.

  6. investigating

    Audit log events are delayed by up to 100 minutes, and Actions workflow status updates continue to experience extended delays. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring recovery and will share further updates as we learn more.

  7. investigating

    Pull Requests is operating normally.

  8. investigating

    Audit log events continue to be delayed by up to 110 minutes. Pull request updates have recovered. Actions is recovering, and users should no longer experience impact in Actions. Webhook deliveries continue to recover as we work through backlogs. We are monitoring recovery and will share further updates as we learn more.

  9. investigating

    Actions and Webhooks are operating normally.

  10. investigating

    Actions and webhook deliveries have recovered. Audit log events are still delayed by about 100 minutes. We are working through the backlog, and full recovery will take some time. We will share more updates as recovery continues.

  11. investigating

    Pull request updates, Actions, and webhook deliveries have recovered. Audit log events are recovering and delays are decreasing, but we expect the remaining backlog to take approximately two to three hours to clear. We’re monitoring recovery and will share further updates.

  12. investigating

    Audit log events are recovering, with delays now below 90 minutes as the remaining backlog is processed. We’ve reduced load on the event-streaming platform and are monitoring recovery. Further updates will follow.

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.

How the archive is checked, and what it cannot prove.

Rewindex captured this after the last checkpoint; the next run () commits it.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

GitHub profileRecent GitHub incidentsMarkdownJSON