What vendors changed

HHexProjects not loading

Hex projects were not loading on the Main site in a critical incident starting 2025-01-08 21:00 UTC and resolved at 2025-01-08 22:41 UTC, lasting 1 h 40 min. Hex reported implementing a fix at 01-08 21:08 UTC and monitoring the results.

CCriblCloud portal not accessible

CCensusCensus unavailable. 503 Error page from the Web.

Census reported a major outage affecting the Sync Management UI and Sync Engine, with a 503 error page, from 2025-01-07 20:53 UTC to 2025-01-07 21:28 UTC (35 min). An infrastructure upgrade inadvertently removed services; all services recovered, and queued syncs began starting at 2025-01-07 21:18 UTC.

IIterableIntermittent API errors

Iterable reported a major incident affecting Global API Success, with intermittent partial API outages lasting several minutes. The incident started at 2025-01-07 19:56 UTC and was resolved at 2025-01-07 22:10 UTC, lasting 2 h 13 min. Engineers implemented a fix and reported no further API outages.

Metabaseand MMetabase CloudMetabase Store unavailable

Metabase reported a major outage of the Metabase Store, starting and resolving at 2025-01-06 19:30 UTC (recorded duration: 0 min). Its resolution update said the store was intermittently unavailable for an hour due to a problem during a store update, conflicting with the recorded duration.

CCloseApp UI unavailable

Close’s critical Application UI incident ran from 2025-01-06 17:49 UTC to 2025-01-06 19:47 UTC (1 h 58 min). The postmortem reports Web UI unavailability from 2025-01-06 17:33 UTC to 2025-01-06 17:48 UTC (15 minutes). Close cited a deployment-process change and said the issue was reverted.

PrismaAccelerate partial disruption affecting new query engine starts

IIterableEvent-Triggered Journeys Delays Ingesting New Users

Iterable reported major delays ingesting new users into Event-Triggered Journeys from 2024-12-31 14:29 UTC to 2024-12-31 18:25 UTC, lasting 3 h 55 min. Errors in workflow-entrance-trigger pods caused a backlog; Iterable updated the service and scaled up pods, then confirmed the backlog was drained. Scheduled Journeys and API Triggered Journeys were unaffected.

JFrogJFrog platform services outage affecting a subset of customers in Azure US East region due to infrastructure issues

JFrog reported a critical platform services outage affecting a subset of customers in Azure US East (Virginia), attributed to infrastructure issues. It started at 2024-12-31 02:41 UTC and lasted 5 h 22 min, affecting Artifactory, Distribution, Pipelines, Security services, Webhook and Web UI. A fix was implemented; the incident was resolved at 2024-12-31 08:03 UTC.

RRudderStackHigh rate of 5xx in ingestion of events for starter and growth tier in US region

RudderStack reported a major incident with high rates of 5xx errors during event ingestion for starter and growth tiers in the US region. It started at 2024-12-27 22:55 UTC and was resolved at 2024-12-27 23:15 UTC, lasting 19 min; the vendor confirmed ingestion was back to normal.

SSynthesiaVideo Generation and Editor Issues

ContentstackContentstack - Azure NA - Network Outage

JFrogJFrog platform services outage affecting a subset of AWS regions due to AWS infrastructure issues

JFrog reported a critical platform services outage due to AWS infrastructure issues, affecting Artifactory, Distribution, Pipelines, security services, Webhook and Web UI in a subset of AWS regions. The outage started at 2024-12-22 13:08 UTC and was resolved at 2024-12-22 14:38 UTC.

SSambaNovaLlama 3.1 8B is having intermittent timeouts

JFrogJFrog Xray services outage affecting a subset of customers in Azure EU West

JFrog reported a major Xray services outage affecting a subset of customers in Azure EU West (Netherlands), starting 2024-12-20 03:24 UTC and resolved at 2024-12-20 03:51 UTC after 27 min. A fix was implemented by 2024-12-20 03:38 UTC; no cause was stated.

FFrontAutomated emails from Front are delayed - notifications, invites to your teammates, calendar events, etc.

Front reported a major incident delaying automated emails, including notifications, teammate invites and calendar events, from 2024-12-18 19:33 UTC to 2024-12-18 20:41 UTC (1 h 8 min). Front worked with its provider to remediate the issue and marked it resolved.

New RelicUI Irregularities

New Relic reported a major incident affecting US UI access, with errors or pages not loading; data ingest and alert notifications were unaffected. The incident started at 2024-12-18 09:18 UTC and was resolved at 2024-12-18 09:47 UTC, lasting 29 min. New Relic reported services returned to normal but stated no cause or specific fix.

npmPackage Install failures due to "304 Not Modified" responses in Mumbai, India region

PostmanFailures in Importing Collections and APIs in a workspace

Postman reported a major outage affecting collection and API imports in workspaces, including collection import flows and API Builder, on its browser and desktop platforms, with US components listed. It started at 2024-12-17 10:38 UTC and was resolved at 2024-12-17 12:13 UTC, lasting 1 h 35 min; no cause or fix was stated.

RRudderStackIncorrect MIME Type Headers for JavaScript SDK Plugin Artifacts

RudderStack reported a major incident: incorrect CDN MIME type headers (binary/octet-stream) caused modern browsers to reject JavaScript SDK plugin scripts, preventing plugin loading and losing instrumented events to RudderStack and device mode destinations. The incident started and was marked resolved at 2024-12-17 09:45 UTC, with a recorded duration of 0 min; impact varied across customers.

ngrokDashboard SSO logins are failing

SubstackElevated Error Rates

Substack reported a major outage with elevated error rates on its websites and mobile application, starting 2024-12-13 16:21 UTC and resolved at 2024-12-13 16:50 UTC (computed duration: 28 min). Services were fully available by 16:28 UTC; listed components included Email delivery, Push Notifications, Subscription invoice processing, Custom domains, Substack.com, Substack publications and Writer dashboards.

PyPI (Python Infrastructure)Infrastructure Outage

PulumiDegraded performance in Pulumi Deployments

Pulumi Deployments experienced increased error rates for deployments building docker images that rely on dockerhub, with impact rated major. The incident started at 2024-12-12 22:03 UTC and was resolved at 2024-12-12 23:57 UTC, lasting 1 h 53 min; no cause or fix was stated.

PulumiPulumi deployments are failing

Pulumi deployments were failing in a major-impact incident starting 2024-12-12 20:58 UTC, lasting 21 min and resolved at 2024-12-12 21:20 UTC. Pulumi reported deploying a fix and monitoring; no cause was stated.

TrelloUsers aren't able to create new repeats or run existing ones when using the Card Repeater Power-Up

Trello’s Card Repeater Power-Up had a major outage: users couldn’t create new repeats, and existing repeats weren’t executing. It lasted 2 h 32 min, from 2024-12-11 15:27 UTC to 2024-12-11 17:59 UTC, affecting API and Trello.com. An initial fix failed; a subsequent fix restored existing repeats and creation of new ones.

ngrokUnable to upgrade to pro or enterprise plans

TypeformTypeform Platform - Submitting Forms

Typeform reported a major incident affecting form-response submissions, starting at 2024-12-10 17:41 UTC and resolved at 2024-12-10 17:54 UTC, with a computed duration of 12 min. Users may have been impacted when submitting responses; Typeform said a fix was implemented but did not state a cause.

Jupyter (Anaconda)Errors when logging into pyscript.com

A major incident caused pyscript.com login attempts to potentially return "profile - username - Input should be a valid string." It started at 2024-12-09 18:05 UTC and was resolved at 2024-12-09 21:06 UTC, lasting 3 h 1 min. A fix was implemented at 20:04 UTC; no cause was stated.

ngrokGetting a high rate of ERR_NGROK_503 in the EU region - No available reserved addresses

PrismaAccelerate disruption affecting new query engine starts

New RelicAPM UI Irregularities

New Relic reported a major incident affecting US APM and UI, with some customers unable to see data in APM monitoring/tools. The incident started at 2024-12-05 09:38 UTC and was resolved at 2024-12-05 10:38 UTC (computed duration: 59 min); the final update separately reported customer impact between 07:20 UTC and 10:25 UTC on Dec 05.

Jupyter (Anaconda)Elevated error rates on repo.anaconda.com

A major incident affected some repo.anaconda.com paths and repo.anaconda.com/pkgs, with elevated errors reported from 2024-12-04 16:05 UTC. The recorded incident began 2024-12-05 08:34 UTC and lasted 5 h 44 min; full functionality returned at 2024-12-05 13:50 UTC. An infrastructure provider identified an issue and was deploying a fix.

AsanaAsana unavailable

Asana reported a critical outage affecting app, API, mobile, automation and webhook components, including EU services, from 2024-12-03 00:09 UTC to 2024-12-03 00:55 UTC (46 min). A deployment-related internal networking error left stale connections, causing European data-center requests to fail; a fix was implemented at 00:22 UTC.

PPandaDocIncident notification

PandaDoc reported a critical file-upload outage affecting PDF, DOCX, CSV, image files and other uploads, with US & Global components listed. It started at 2024-12-02 17:44 UTC and was fully resolved at 2024-12-02 18:35 UTC, lasting 51 min; PandaDoc reported uploads restored at 18:22 UTC.

QuickBooks OnlineWe’re working to fix problems with QuickBooks Merchant Account

A major outage prevented some QuickBooks Merchant Account customers in the United States from processing credit cards inside QuickBooks Desktop from 2024-12-02 17:18 UTC to 2024-12-02 20:01 UTC, lasting 2 h 42 min. QuickBooks suggested processing payments through the Merchant Service Center as a workaround, then implemented a fix and confirmed resolution.

PrismaAccelerate partial disruption in us-east-1

IimgixElevated SSL/TLS expiration errors for custom domains

DDailyNetworking issues

IIterableWeb Application Errors

Iterable reported a major incident involving API and web application errors, with delays to sends and journey processing. It started at 2024-11-24 18:16 UTC and was resolved at 2024-11-25 17:09 UTC, lasting 22 h 52 min; engineering applied mitigations and reported recovery in the web application and API endpoints.

GustoCustomers are experiencing slowness and errors when accessing Gusto

Gusto reported a critical incident preventing customers from accessing their accounts and affecting API performance. It started at 2024-11-22 19:09 UTC and was resolved at 2024-11-22 20:57 UTC, lasting 1 h 48 min. Gusto identified the issue and implemented a fix; no cause was stated.

NNordLayerPossible login issues with applications and Browser Extension

NordLayer reported a major login incident affecting its applications and Browser Extension, with increased failed login attempts. It started at 2024-11-20 15:49 UTC and was resolved at 2024-11-20 17:23 UTC, lasting 1 h 33 min. A fix was implemented at 16:48 UTC; no cause was stated.

CockroachDBElevated error rate for some Standard and Basic clusters

CockroachDB reported a major incident with elevated error rates on isolated Standard and Basic clusters from 2024-11-19 16:48 UTC to 2024-11-19 18:36 UTC, lasting 1 h 48 min. Engineering reported identifying and resolving the root cause, without specifying it, and confirmed resolution for all affected clusters.

FFront[us-west-2] Increased failure rate on public API

Front reported a major public API incident in us-west-2, with increased 5xx errors potentially affecting sending and receiving messages on some channels. It ran from 2024-11-18 20:03 UTC to 2024-11-18 22:18 UTC, lasting 2 h 15 min. Errors subsided before resolution; no root cause or fix was stated.

PulumiMajor outage in Pulumi Cloud API

Pulumi reported a critical outage affecting the Pulumi Cloud API and Deployments, starting at 2024-11-18 19:40 UTC and resolved at 2024-11-18 19:59 UTC; the computed duration was 18 min. A fix was implemented by 19:50 UTC, followed by monitoring.

FFireboltUnable to start Engines on FB 2.0

Firebolt reported a critical FB 2.0 outage across all regions: engines could not start or Autostop, and inserts failed on running engines. The incident started at 2024-11-14 19:46 UTC, with symptoms reported since 18:26 UTC; its recorded duration was 23 min. Reverting a recent network change restored service at 20:03 UTC; it was marked resolved at 20:10 UTC.

UpstashPerformance degradation on QStash

QStash had a major incident affecting EU-CENTRAL-1, with extremely high event-processing latency for all users and duplicate deliveries for some customers. It ran from 2024-11-13 14:14 UTC to 2024-11-13 17:27 UTC, lasting 3 h 12 min. Upstash attributed the degradation to sudden, extreme server load and reported implementing a fix before resolution.

HHexPlatform instability

Hex reported critical platform instability affecting kernel execution and site access, starting 2024-11-12 16:18 UTC and resolved 2024-11-12 19:08 UTC (2 h 50 min). An unintended effect of a recent database migration caused a backlog and impacted platform performance; Hex reported implementing a fix and resetting kernels.

TypeformTypeform Renderer - Forms intermittently unavailable

Typeform reported a major outage causing intermittent failures when opening/loading forms and submitting responses, starting 2024-11-08 21:28 UTC. The incident lasted 53 min and was resolved at 2024-11-08 22:22 UTC after a fix was implemented; no cause was stated.