Struck in red is what Okta removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
AddedTitle#1
Before: not present
After: Issues with Custom Domain in Cell OK8
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2025-01-14 17:31 UTC
AddedResolved time#5
Before: not present
After: 2025-01-14 00:00 UTC
AddedAffected components#6
Before: not present
After: Cell okta.com:8, Core Platform / Core Service
AddedUpdateresolved#7
Before: not present
After: At 1/14/2025 9:16 AM PST, the Core Identity team became aware of an issue with Custom Domains affecting customers in OK8. During this time, users may have experienced issues accessing resources powered by Okta custom domains. This issue has been resolved. Okta took corrective action to resolve the service interruption. The service was fully restored at 9:33 AM PST.
Root cause information:
We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.
Detection and Impact:
On January 14th at 9:22AM PT Okta internal monitoring alerts indicated errors in loading custom domains in OK Cell 8.
During this time customers who utilize customized domains in OK Cell 8 experienced errors accessing the service.
Root Cause Summary:
In order to resolve an issue which was caught by our internal monitoring during a change the previous evening, Okta engineering staff was redeploying the edge servers. Due to a bug in the operating procedure used, the service became momentarily unavailable until corrective actions were taken to fully restore the service.
Remediation Steps:
At 9:24AM PT, Okta Engineering quickly identified the issue and resolved it by placing healthy servers in service.
Preventative Actions:
Okta has updated the operating procedure and improved the tooling used for managing custom domain services. To ensure this does not happen again, Okta is enhancing current automated testing of this process.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.