# Grain: Recording Instability

> Grain status incident “Recording Instability” on 2023-07-10: impact none (vendor’s label), 0 min, 2 update(s). Archived by Rewindex.

Canonical: https://rewindex.com/company/grain/incidents/z74f6kbydvpz
Last updated: 2023-08-02T14:53:30Z

- Impact (vendor’s label): none
- Status: postmortem
- Started: 2023-07-10 05:00 UTC · Resolved: 2023-07-10 05:00 UTC · Duration: 0 min
- Updates: 2
- [Vendor status page: https://stspg.io/p90blsl6fvtb](https://stspg.io/p90blsl6fvtb)
- [Company: Grain](https://rewindex.com/company/grain.md)

## Timeline of updates

- **resolved** 2023-07-10 05:00 UTC: During the peak hours of 07/10 and 07/11, the Grain recording bot service experienced instability due to increased load, causing the recording bot to fail to join meetings or correctly capture recordings for some customers. After mitigations were deployed, monitoring shows that the system has stabilized. Further details on the outage are being compiled.
- **postmortem** 2023-08-02 14:53 UTC: Recording failures due to bot network timeouts \* On Monday and Tuesday of the week, a significant number of bots failed to join calls due to DNS requests timing out. \* The root cause was Kubernetes cluster DNS servers being overloaded by the volume of incoming requests; specifically the DNS network traffic at peak load exceeded the inbound network bandwidth limits of the EC2 instances running the DNS servers. \* This caused a subset of DNS packets to be dropped at the AWS network interface layer, resulting in the timeouts. \* This issue emerged on 07/10/2023 as previous fix to improve the speed of scheduled bots joining calls caused a large spike in traffic within a few hundred millisecond window as opposed to spreading the bots out more gradually. Two major mitigations have been made: 1. The bots to no longer rely on the Kubernetes cluster DNS and instead rely on the AWS Route53 resolver which is significantly more scaleable and able to handle the peak load. 2. Improved monitoring and alerts are in place for inbound and outbound network traffic quotas on EC2 instances to allow for better responsiveness for any future issues.

## Cite this

Status incident (vendor impact label: none) posted by Grain, started 2023-07-10 05:00 UTC, resolved after 0 min, 2 update(s).

- Canonical URL: https://rewindex.com/company/grain/incidents/z74f6kbydvpz
- Citation: Grain, “Recording Instability” (2023-07-10 05:00 UTC). Archived by Rewindex: https://rewindex.com/company/grain/incidents/z74f6kbydvpz (CC BY 4.0).
- Markdown link: `[Grain: Recording Instability (2023-07-10)](https://rewindex.com/company/grain/incidents/z74f6kbydvpz)`


---
Canonical: https://rewindex.com/company/grain/incidents/z74f6kbydvpz · Markdown: https://rewindex.com/company/grain/incidents/z74f6kbydvpz.md · JSON: https://rewindex.com/api/v1/companies/grain/incidents/z74f6kbydvpz · Data licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) · Generated 2026-11-04T15:32:47.553Z