Changes / Grain / Application instability

Grain: Major incident: Application instability

Summary

Grain Web App experienced major application instability with a high rate of errors after a recent code deploy, from 2024-01-19 17:18 UTC to 2024-01-19 18:43 UTC (1 h 25 min). The team issued a hot patch and marked the incident resolved after observing no further instability.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 74%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Application instability
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2024-01-19 17:18 UTC
AddedResolved time#5
Before: not present
After: 2024-01-19 18:43 UTC
AddedAffected components#6
Before: not present
After: Grain Web App
AddedUpdateinvestigating#7
Before: not present
After: The team is observing a high rate of errors after a recent code deploy and we are now investigating the issue. This is causing instability in the application.
AddedUpdatemonitoring#8
Before: not present
After: The team has identified the issue and issued a hot patch, metrics show the issue is clearing up.
AddedUpdateresolved#9
Before: not present
After: We are marking this as resolved as we have not observed further instability.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Application instability (this change)
Grain Web App experienced major application instability with a high rate of errors after a recent code deploy, from 2024-01-19 17:18 UTC to 2024-01-19 18:43 UTC (1 h 25 min). The team issued a hot patch and marked the incident resolved after observing no further instability.
Major incident

All Grain changes → · Grain profile · Markdown · JSON