# Render: Incident updated: Builds, deploys, and queued deploys are not starting in all regions

> Incident updated: Builds, deploys, and queued deploys are not starting in all regions. Detected 2026-10-22 by Rewindex: what changed on Render’s status page, with the before/after diff.

Canonical: https://rewindex.com/change/r299ca34e521024
Last updated: 2026-10-22T18:24:11Z

**Summary:** _summary pending_
- Kind: Incident update · Label: Normal progress (rule)
- Changed: 2026-10-22 18:24 UTC · Detected: 2026-10-22 18:24 UTC
- [Source: https://stspg.io/qb9l5v44x73f](https://stspg.io/qb9l5v44x73f)
- [Company: Render](https://rewindex.com/company/render.md)
- Versions: before incident\_version:203/8tsd93r1wbmz@e4d0237a449d → after incident\_version:203/8tsd93r1wbmz@4025e5fcab59
- Change 2 of 2 for this incident

Incident page (full timeline): [Builds, deploys, and queued deploys are not starting in all regions](https://rewindex.com/company/render/incidents/8tsd93r1wbmz.md)
## What changed

**1. Changed: Status**

```diff
- resolved
+ postmortem
```

**2. Added: Update** (postmortem) · 2026-10-22 18:20 UTC

```diff
+ Summary On October 14, 2026, between 12:12 and 15:08 PDT (19:12 - 22:08 UTC), builds and deploys across all regions failed to start, stalled, or were significantly delayed during two periods of impact totaling about 2 hours and 20 minutes. The cause was a database schema migration run as part of a planned version upgrade of our internal job scheduler that coordinates builds, deploys, and other critical platform operations. The migration led to high load and a longer-than-expected recovery time of this system’s database. We will change how we perform upgrades and modifications of this system in a highly available fashion in the future so that a similar change cannot cause an outage again. Impact There were two periods of impact. During this time, live user services were not affected, but builds and deploys across all regions failed to start, failed to finish, or were significantly delayed. Period 1: 12:12 - 12:40 PDT (19:12 - 19:40 UTC), about 28 minutes New builds and deploys did not start. Deploys in progress stalled. Some requests to create a deploy through the Dashboard, API, and CLI returned errors (HTTP 500). Partial recovery: 12:40 - 13:18 PDT (19:40 - 20:18 UTC) Builds and deploys ran, but deploys took about 20 seconds to start instead of a few hundred milliseconds. Period 2: 13:18 - 15:08 PDT (20:18 - 22:08 UTC), about 1 hour 50 minutes Builds and deploys were delayed, stalled in progress (including in pre-deploy), or failed. From 13:51 PDT, almost no new builds were able to start. Some queued deploys did not start after recovery, and cancelling them through the API had no effect. Users needed to retry these deploys. Root Cause Render uses an internal job scheduler system to coordinate builds, deploys, and other critical platform operations. We were rolling out improvements to how it distributes work, and began a planned version upgrade of this system in production after a successful practice run in a test environment. The upgrade included a database schema migration that locked a large table and drove database CPU utilization to 100%. This database stores both the system’s core job state and secondary data used for search and reporting. The system could not read or write job state reliably, so builds and deploys stopped or slowed. Engineers were alerted to the issue quickly and reverted the upgrade. However, three factors extended the impact: A pre-upgrade step to create a database clone for rollback could not be completed due to an error from the underlying cloud provider, so a fast rollback was not available. The migration continued to run on the database after the client performing the upgrade was stopped and the upgrade reverted. It was found and terminated at 13:49 PDT, about 1.5 hours after it started. An attempt to add capacity to the database failed due to an error from the underlying cloud provider. Timeline All times are PDT on 2026-10-14 (UTC-7). 12:13 - The upgrade is started, including the database schema migration. Database load rises and builds and deploys stop starting. 12:24 - Impact to builds and deploys identified. Incident escalated. 12:33 - Status page incident opened. 12:34 - Engineers revert the upgrade and stop the client performing the schema migration. 12:37 - Job scheduler system recovers. Builds and deploys resume, with slower deploy start times. End of Period 1. 13:17 - Engineers restart one unhealthy instance of the system to rebalance load. Database connections and errors increase across the system. Start of Period 2. 13:37 - Free-tier deploys paused to reduce load. 13:49 - Engineers find the schema migration still running on the database and terminate it. 14:16 - Database restarted. An attempt to add database capacity fails because of an error from the underlying cloud provider. 14:27 - Remaining components of the system restarted. 14:30 - Builds and deploys start to recover, with slower than usual build and deploy times as the backlog is processed. 15:08 - Deploy start times return to normal and database load returns to pre-incident levels. End of Period 2. 15:11 - Free-tier deploys and spin-ups gradually re-enabled. 15:18 - Incident resolved. Mitigations Completed during the incident Reverted the job scheduler component upgrade and configuration change (12:34 PDT). Paused free-tier deploys and spin-ups to reduce load, then re-enabled them gradually after recovery. Terminated the schema migration that was still running on the database (13:49 PDT). Restarted the database and the job scheduler system to restore normal operation. Paused the upgrade in production until the upgrade process is changed. Ongoing (started during the incident) This database is being upgraded to support cross-zone High Availability. Planned Implement a new upgrade and deployment practice for this component: we will implement blue-green deploys to avoid upgrading this system under load in the future. Adjust the underlying storage mechanisms for the job scheduler to be better suited to this workload type, after we have implemented blue-green deploys.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-10-14 19:32 UTC | [Major incident: Builds, deploys, and queued deploys are not starting in all regions](https://rewindex.com/change/if2269212c75fab.md) | outage |
| 2026-10-22 18:24 UTC | Incident updated: Builds, deploys, and queued deploys are not starting in all regions (this change) | Normal progress |

Previous: [change](https://rewindex.com/change/if2269212c75fab.md) · All Render changes: [timeline](https://rewindex.com/company/render/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._

## Cite this

Incident updated: Builds, deploys, and queued deploys are not starting in all regions on Render’s status page, changed 2026-10-22 18:24 UTC (change 2 of 2).

- Canonical URL: https://rewindex.com/change/r299ca34e521024
- Citation: Render, “Incident updated: Builds, deploys, and queued deploys are not starting in all regions” (2026-10-22 18:24 UTC). Archived by Rewindex: https://rewindex.com/change/r299ca34e521024 (CC BY 4.0).
- Markdown link: `[Render: Incident updated: Builds, deploys, and queued deploys are not starting in all regions (2026-10-22)](https://rewindex.com/change/r299ca34e521024)`


---
Canonical: https://rewindex.com/change/r299ca34e521024 · Markdown: https://rewindex.com/change/r299ca34e521024.md · JSON: https://rewindex.com/api/v1/changes/r299ca34e521024 · Data licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) · Generated 2026-11-11T04:20:59.506Z