Changes / Statuspage / Intermittent errors while accessing public Statuspages
Statuspage: Major incident: Intermittent errors while accessing public Statuspages
Summary
Statuspage recorded major intermittent public-page errors starting 2023-10-28 07:36 UTC, resolved at 2023-10-28 08:01 UTC (24 min). The postmortem reports all functionality affected in all regions from 06:00 UTC to 07:45 UTC on October 28, 2023, due to a database performance issue during scheduled database maintenance.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 89%)Changed Detected Source: mstspg.co (opens in new tab)
What changed
Before: not present
After: Intermittent errors while accessing public Statuspages
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-10-28 07:36 UTC
Before: not present
After: 2023-10-28 08:01 UTC
Before: not present
After: Hosted Pages - HTTP Pages, Hosted Pages - HTTPS Pages, Hosted Pages - Public API, HTTP Pages, HTTPS Pages, Public API
Before: not present
After: We are currently seeing intermittent errors in viewing public Statuspages. We are investigating this problem and will provide updates shortly
Before: not present
After: Update: We have fixed the issue and are monitoring actively
Before: not present
After: Issue is now resolved and everything is back to normal working state.
Before: not present
After: ### **SUMMARY**
From 06:00 UTC to 07:45 UTC on October 28, 2023, Atlassian customers using Statuspage had intermittent issues with all Statuspage functionality. The event occurred due to a database performance issue during a [scheduled database maintenance](https://metastatuspage.com/incidents/s21b66328h9j). This impacted customers in all regions. The incident was detected within one minute by monitoring the upgrade process and mitigated by rolling back to a known good snapshot which put Statuspage systems into a known good state. The total time to resolution was about one hour and 45 minutes.
### **IMPACT**
The overall impact was between 06:00 UTC and 07:45 UTC October 28, 2023. This incident affected Statuspage customers from all regions and caused intermittent backend errors on all Statuspage activity including viewing pages, adding subscribers, and creating/updating events. We performed a rollback operation during recovery to return to a known good state.
### **ROOT CAUSE**
The issue was caused by database performance issues after a routine database maintenance and upgrade. As a result, our backends returned intermittent errors to several user requests.
### **REMEDIAL ACTIONS PLAN & NEXT STEPS**
We take the utmost care to provide a highly reliable service. We will pursue several preventive measures to ensure that this situation does not occur in the future, including:
* Fixing the cause of the performance issues before future upgrades; and
* Improving our testing process for database upgrades to catch potential performance issues.
We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.
Thanks,
Atlassian Customer Support
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Intermittent errors while accessing public Statuspages (this change) Statuspage recorded major intermittent public-page errors starting 2023-10-28 07:36 UTC, resolved at 2023-10-28 08:01 UTC (24 min). The postmortem reports all functionality affected in all regions from 06:00 UTC to 07:45 UTC on October 28, 2023, due to a database performance issue during scheduled database maintenance. | Major incident |
All Statuspage changes → · Statuspage profile · Markdown · JSON