rewindex

CompaniesStatuspageIncidents

Statuspage: Issues With Login

MinorStatus: postmortemStarted Resolved (18 min)3 updatesmstspg.co (opens in new tab)

Affected components: Admin User+PassAuthentication - Admin User+Pass

Timeline of updates

  1. monitoring

    We have identified and mitigated an issue with users logging in to Statuspage starting at 6:05am PST and ending at 6:46am PST and are monitoring the results.

  2. resolved

    This incident has been resolved.

  3. postmortem

    SUMMARY

    On March 14, 2022, between 01:05pm and 01:47pm UTC, some Atlassian customers were unable to login to our products including Trello and Statuspage, and could not access some services including the ability to create support tickets. The underlying cause was a newly introduced configuration data store that did not scale up properly due to a misconfiguration of autoscaling.

    The incident was detected by Atlassian's automated monitoring system and mitigated by disabling the use of the new configuration datastore which put our systems into a known good state. The total time to resolution was approximately 42 minutes.

    IMPACT

    The overall impact was between March 14 2022, 01:05 PM UTC and March 14, 2022, 01:47 PM UTC across seven products and services. The bug impacted several of the key dependent services which resulted in an outage for end users, leading to failed logins across the following products and services:

    ROOT CAUSE

    The issue was caused by an underlying configuration data store based on AWS DynamoDB failing to scale up. During post-setup fine-tuning it was identified that initial values for the read capacity units (RCUs) and write capacity units (RCUs) were over-provisioned. As a result a decision was made to decrease them however the resulting values proved to be insufficient to handle the increased traffic in our system.

    REMEDIAL ACTIONS PLAN & NEXT STEPS

    We're prioritizing the following improvement actions to avoid repeating this type of incident:

    • Fix the configuration so that the new configuration data store dynamically scales-up regardless of the size of the incoming traffic.
    • Conduct more thorough capacity planning and load testing.
    • Improve the resilience of the system by adding fallbacks to our secondary data store.

    We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

    Thanks,

    Atlassian Customer Support

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on mstspg.co (opens in new tab).

Cite this

Status incident (vendor impact label: minor) posted by Statuspage, started 2022-03-14 14:10 UTC, resolved after 18 min, 3 update(s).

Title
Issues With Login
Vendor
Statuspage
Date
URL
https://rewindex.com/company/statuspage/incidents/70l22zqq0090

Statuspage profileRecent Statuspage incidentsAll outages on March 14, 2022Statuspage in March 2022MarkdownJSON