Companies/Grain/Incidents

Grain: Recording Instability

Timeline of updates

1. resolved2023-07-10 05:00 UTC

   During the peak hours of 07/10 and 07/11, the Grain recording bot service experienced instability due to increased load, causing the recording bot to fail to join meetings or correctly capture recordings for some customers. After mitigations were deployed, monitoring shows that the system has stabilized. Further details on the outage are being compiled.

2. postmortem2023-08-02 14:53 UTC

   Recording failures due to bot network timeouts

   • On Monday and Tuesday of the week, a significant number of bots failed to join calls due to DNS requests timing out.
   • The root cause was Kubernetes cluster DNS servers being overloaded by the volume of incoming requests; specifically the DNS network traffic at peak load exceeded the inbound network bandwidth limits of the EC2 instances running the DNS servers.
   • This caused a subset of DNS packets to be dropped at the AWS network interface layer, resulting in the timeouts.
   • This issue emerged on 07/10/2023 as previous fix to improve the speed of scheduled bots joining calls caused a large spike in traffic within a few hundred millisecond window as opposed to spreading the bots out more gradually.

   Two major mitigations have been made:

   1. The bots to no longer rely on the Kubernetes cluster DNS and instead rely on the AWS Route53 resolver which is significantly more scaleable and able to handle the peak load.
   2. Improved monitoring and alerts are in place for inbound and outbound network traffic quotas on EC2 instances to allow for better responsiveness for any future issues.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Grain profileRecent Grain incidentsAll outages on July 10, 2023Grain in July 2023MarkdownJSON