Changes / WorkOS / Degraded performance for Directory Provisioning for Authkit
WorkOS: Major incident: Degraded performance for Directory Provisioning for Authkit
Summary
WorkOS reported a major Directory Provisioning for AuthKit degradation starting 2026-06-22 21:36 UTC, lasting 2 days 7 h and resolved at 2026-06-25 05:12 UTC. Directories created before June 22, 21:36:17 UTC had delayed downstream user and organization membership changes. WorkOS attributed the issue to an internal production data migration; remaining out-of-sync user records were restored.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 90%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Degraded performance for Directory Provisioning for Authkit
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-06-22 21:36 UTC
Before: not present
After: 2026-06-25 05:12 UTC
Before: not present
After: AuthKit, Core Services - AuthKit, Core Services - Directory Sync, Directory Sync
Before: not present
After: We are seeing a delay in Directory Provisioning for Authkit, for any directories created before June 22, 21:36:17 UTC. Any directories created after this time are working as expected.
Before: not present
After: The underlying issue that has lead to a delay in provisioning has been identified, and a fix is being prepared
Before: not present
After: We have corrected the affected directories, and they are back to a provisioning status.
Before: not present
After: Provisioning has caught up for user creation, and new directory activity is syncing normally. We are still working through and monitoring the remaining synchronization catch-up, which is not yet fully complete. Another update to follow.
Before: not present
After: Synchronization has continued to catch up. We are continuing to monitor the few out of sync changes that are remaining.
Before: not present
After: The remaining out-of-sync user records are now back in sync.
Before: not present
After: ## **Summary**
Between June 22, 2026 at 21:36 UTC and June 25, 2026 at 05:12 UTC, Directory Provisioning for AuthKit was interrupted. During this period, directory updates continued to be received and processed by WorkOS, but downstream AuthKit user and organization membership changes were not applied as expected.
The issue was caused by an internal data migration being run in production with parameters that unintentionally changed the AuthKit provisioning state for existing customer directories. Once identified, we restored the affected directory provisioning state, backfilled missed user provisioning updates, and deactivated memberships that had not been deactivated during the incident window. No directory events were lost or dropped during this period; however, events received during the incident period were not immediately propagated to AuthKit. Directories created after the migration were unaffected.
## Background
Directory Provisioning for AuthKit works in two steps: WorkOS receives directory events from a customer's identity provider, then propagates those events downstream to create or update AuthKit users and organization memberships. That second step only happens if the directory is marked as eligible for downstream provisioning. If that eligibility is disabled, WorkOS continues receiving directory events normally, but the corresponding AuthKit updates are not applied.
## What Happened
On June 22, 2026 at 21:36 UTC, an internal data migration was executed through an administrative migration tool. The migration changed the provisioning status for customer directories in a way that unintentionally disabled Directory Provisioning in AuthKit for affected customers.
As a result:
* New directory users created during the incident window were not consistently provisioned into AuthKit.
* Some directory user updates, including attributes and roles, were not propagated to corresponding organization memberships.
* Some users who were removed or deactivated in the directory were not fully deactivated in AuthKit during the incident period. This meant that, for the duration of the incident, some users who should have lost access to an organization may have retained it. Based on our review, we found no evidence that this residual access was exercised or misused during the incident window.
The incident was reported on June 24, 2026 after customer reports of provisioning failures, approximately 42 hours after the migration ran. We did not have sufficient monitoring in place at the time that was sensitive enough to detect an abnormal drop in volume for Directory Provisioning in AuthKit, which is why detection relied on customer reports rather than internal alerting. Our team investigated the behavior, identified the migration as the cause, and immediately worked to restore affected directory provisioning state.
We then performed additional remediation to address downstream effects from the incident window. This included provisioning missed users, refreshing affected membership attributes and roles, and deactivating stale memberships that should no longer have been active.
## Customer Impact & Recommended Actions
Customers with Directory Provisioning enabled for AuthKit during the incident window may have experienced delayed user provisioning, attribute/role updates, or deactivation between June 22, 2026 21:36 UTC and June 25, 2026 05:12 UTC. All known gaps from this window have since been backfilled as part of our remediation.
If you have reason to believe your AuthKit users or organization memberships were affected in a way not addressed by this remediation, please reach out to our support team so we can investigate your specific environment.
## **Timeline**
* **June 22, 2026 21:36 UTC** – An internal data migration unintentionally disabled Directory Provisioning for affected directories.
* **June 24, 2026 15:49 UTC** – The incident was reported after customer provisioning failures were identified.
* **June 24, 2026 17:23 UTC** – We restored provisioning status for all affected directories, allowing new provisioning activity to resume.
* **June 24, 2026 23:30 UTC** – We completed the first remediation pass to propagate provisioning updates missed during the incident period.
* **June 25, 2026 02:10 UTC** – We completed remediation to refresh affected role and custom attribute data.
* **June 25, 2026 05:12 UTC** – We completed deactivation remediation and considered the incident fixed.
## **Remediation**
### **Immediate Fixes**
We took the following steps to restore service and correct affected data:
* Restored the provisioning status for affected directories to resume Directory Provisioning in AuthKit.
* Backfilled user creations, updates, and deactivations made by the identity provider during the incident window that had not been propagated to AuthKit users and organization memberships.
* Removed the unsafe migration path that caused the incident so it could not be run again.
### **Near-Term Improvements**
Additionally, we are making the following changes to reduce the likelihood and impact of similar incidents:
* **Improved monitoring and detection:** We are adding additional monitoring and alerting for Directory Provisioning in AuthKit behavior, so we can more quickly respond if a similar issue were to occur in the future.
* **Harden internal migration tooling:** We are reviewing access and safeguards for internal migration tools, including stronger controls for destructive operations and clearer scoping requirements.
## **Conclusion**
Reliable user provisioning and deactivation are critical parts of identity infrastructure, and this incident did not meet the standard we hold ourselves to. We regret the impact this had on affected customers and their users.
The incident has been resolved, affected data has been remediated, and we are implementing the controls and monitoring described above to help prevent similar incidents in the future. Please reach out to our team if you have additional questions on the impact to your environments.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Degraded performance for Directory Provisioning for Authkit (this change) WorkOS reported a major Directory Provisioning for AuthKit degradation starting 2026-06-22 21:36 UTC, lasting 2 days 7 h and resolved at 2026-06-25 05:12 UTC. Directories created before June 22, 21:36:17 UTC had delayed downstream user and organization membership changes. WorkOS attributed the issue to an internal production data migration; remaining out-of-sync user records were restored. | Major incident |