CompaniesGrafana CloudIncidents
Grafana Cloud: Increased timeout errors in alert rule evaluations (AWS Germany, prod-eu-west-2)
Affected components: Alerting and SLO - AWS Germany - prod-eu-west-2: Rule EvaluationAWS Germany - prod-eu-west-2: Rule Evaluation
Timeline of updates
- investigating
Since approximately 20:30 UTC on October 20, we have observed an increase in alert rule evaluation errors affecting rules that query Grafana Cloud Prometheus (Mimir) data sources in Cluster AWS Germany.
Our engineering team is actively investigating and working on remediation. - identified
The root cause of the issue has been indentified and we are working on the remediation steps.
- monitoring
Remediation measures have been applied, and we are monitoring the recovery.
- resolved
This incident has been resolved.
Changes Rewindex caught on this incident
- Incident updated: Increased timeout errors in alert rule evaluations (AWS Germany, prod-eu-west-2)
- Incident updated: Increased timeout errors in alert rule evaluations (AWS Germany, prod-eu-west-2)
- Incident updated: Increased timeout errors in alert rule evaluations (AWS Germany, prod-eu-west-2)
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
Rewindex captured this after the last checkpoint; the next run () commits it.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).
Grafana Cloud profileRecent Grafana Cloud incidentsMarkdownJSON