What vendors changed

PulumiPulumi deployments are failing

Pulumi deployments were failing in a major-impact incident starting 2024-12-12 20:58 UTC, lasting 21 min and resolved at 2024-12-12 21:20 UTC. Pulumi reported deploying a fix and monitoring; no cause was stated.

TrelloUsers aren't able to create new repeats or run existing ones when using the Card Repeater Power-Up

Trello’s Card Repeater Power-Up had a major outage: users couldn’t create new repeats, and existing repeats weren’t executing. It lasted 2 h 32 min, from 2024-12-11 15:27 UTC to 2024-12-11 17:59 UTC, affecting API and Trello.com. An initial fix failed; a subsequent fix restored existing repeats and creation of new ones.

ngrokUnable to upgrade to pro or enterprise plans

TypeformTypeform Platform - Submitting Forms

Typeform reported a major incident affecting form-response submissions, starting at 2024-12-10 17:41 UTC and resolved at 2024-12-10 17:54 UTC, with a computed duration of 12 min. Users may have been impacted when submitting responses; Typeform said a fix was implemented but did not state a cause.

Jupyter (Anaconda)Errors when logging into pyscript.com

A major incident caused pyscript.com login attempts to potentially return "profile - username - Input should be a valid string." It started at 2024-12-09 18:05 UTC and was resolved at 2024-12-09 21:06 UTC, lasting 3 h 1 min. A fix was implemented at 20:04 UTC; no cause was stated.

ngrokGetting a high rate of ERR_NGROK_503 in the EU region - No available reserved addresses

PrismaAccelerate disruption affecting new query engine starts

New RelicAPM UI Irregularities

New Relic reported a major incident affecting US APM and UI, with some customers unable to see data in APM monitoring/tools. The incident started at 2024-12-05 09:38 UTC and was resolved at 2024-12-05 10:38 UTC (computed duration: 59 min); the final update separately reported customer impact between 07:20 UTC and 10:25 UTC on Dec 05.

Jupyter (Anaconda)Elevated error rates on repo.anaconda.com

A major incident affected some repo.anaconda.com paths and repo.anaconda.com/pkgs, with elevated errors reported from 2024-12-04 16:05 UTC. The recorded incident began 2024-12-05 08:34 UTC and lasted 5 h 44 min; full functionality returned at 2024-12-05 13:50 UTC. An infrastructure provider identified an issue and was deploying a fix.

AsanaAsana unavailable

Asana reported a critical outage affecting app, API, mobile, automation and webhook components, including EU services, from 2024-12-03 00:09 UTC to 2024-12-03 00:55 UTC (46 min). A deployment-related internal networking error left stale connections, causing European data-center requests to fail; a fix was implemented at 00:22 UTC.

PPandaDocIncident notification

PandaDoc reported a critical file-upload outage affecting PDF, DOCX, CSV, image files and other uploads, with US & Global components listed. It started at 2024-12-02 17:44 UTC and was fully resolved at 2024-12-02 18:35 UTC, lasting 51 min; PandaDoc reported uploads restored at 18:22 UTC.

QuickBooks OnlineWe’re working to fix problems with QuickBooks Merchant Account

A major outage prevented some QuickBooks Merchant Account customers in the United States from processing credit cards inside QuickBooks Desktop from 2024-12-02 17:18 UTC to 2024-12-02 20:01 UTC, lasting 2 h 42 min. QuickBooks suggested processing payments through the Merchant Service Center as a workaround, then implemented a fix and confirmed resolution.

PrismaAccelerate partial disruption in us-east-1

IimgixElevated SSL/TLS expiration errors for custom domains

DDailyNetworking issues

IIterableWeb Application Errors

Iterable reported a major incident involving API and web application errors, with delays to sends and journey processing. It started at 2024-11-24 18:16 UTC and was resolved at 2024-11-25 17:09 UTC, lasting 22 h 52 min; engineering applied mitigations and reported recovery in the web application and API endpoints.

GustoCustomers are experiencing slowness and errors when accessing Gusto

Gusto reported a critical incident preventing customers from accessing their accounts and affecting API performance. It started at 2024-11-22 19:09 UTC and was resolved at 2024-11-22 20:57 UTC, lasting 1 h 48 min. Gusto identified the issue and implemented a fix; no cause was stated.

NNordLayerPossible login issues with applications and Browser Extension

NordLayer reported a major login incident affecting its applications and Browser Extension, with increased failed login attempts. It started at 2024-11-20 15:49 UTC and was resolved at 2024-11-20 17:23 UTC, lasting 1 h 33 min. A fix was implemented at 16:48 UTC; no cause was stated.

CockroachDBElevated error rate for some Standard and Basic clusters

CockroachDB reported a major incident with elevated error rates on isolated Standard and Basic clusters from 2024-11-19 16:48 UTC to 2024-11-19 18:36 UTC, lasting 1 h 48 min. Engineering reported identifying and resolving the root cause, without specifying it, and confirmed resolution for all affected clusters.

FFront[us-west-2] Increased failure rate on public API

Front reported a major public API incident in us-west-2, with increased 5xx errors potentially affecting sending and receiving messages on some channels. It ran from 2024-11-18 20:03 UTC to 2024-11-18 22:18 UTC, lasting 2 h 15 min. Errors subsided before resolution; no root cause or fix was stated.

PulumiMajor outage in Pulumi Cloud API

Pulumi reported a critical outage affecting the Pulumi Cloud API and Deployments, starting at 2024-11-18 19:40 UTC and resolved at 2024-11-18 19:59 UTC; the computed duration was 18 min. A fix was implemented by 19:50 UTC, followed by monitoring.

FFireboltUnable to start Engines on FB 2.0

Firebolt reported a critical FB 2.0 outage across all regions: engines could not start or Autostop, and inserts failed on running engines. The incident started at 2024-11-14 19:46 UTC, with symptoms reported since 18:26 UTC; its recorded duration was 23 min. Reverting a recent network change restored service at 20:03 UTC; it was marked resolved at 20:10 UTC.

UpstashPerformance degradation on QStash

QStash had a major incident affecting EU-CENTRAL-1, with extremely high event-processing latency for all users and duplicate deliveries for some customers. It ran from 2024-11-13 14:14 UTC to 2024-11-13 17:27 UTC, lasting 3 h 12 min. Upstash attributed the degradation to sudden, extreme server load and reported implementing a fix before resolution.

HHexPlatform instability

Hex reported critical platform instability affecting kernel execution and site access, starting 2024-11-12 16:18 UTC and resolved 2024-11-12 19:08 UTC (2 h 50 min). An unintended effect of a recent database migration caused a backlog and impacted platform performance; Hex reported implementing a fix and resetting kernels.

TypeformTypeform Renderer - Forms intermittently unavailable

Typeform reported a major outage causing intermittent failures when opening/loading forms and submitting responses, starting 2024-11-08 21:28 UTC. The incident lasted 53 min and was resolved at 2024-11-08 22:22 UTC after a fix was implemented; no cause was stated.

AsanaIssues in some regions

Asana reported a critical outage affecting API, App, Mobile, Automations and Background Actions, and Webhooks and Event Streams in regions outside the US, including Australia, EU and Japan, due to a network issue. It started at 2024-11-08 20:46 UTC and lasted 21 min; a fix was implemented, and the incident was resolved at 2024-11-08 21:08 UTC.

CCriblMultiple organizations did not upgrade during 4.9.1 cloud release

BitbucketUnable to invite new users due to missing recaptcha token

Bitbucket reported a major incident preventing some Cloud customers from inviting new users through admin.atlassian.com due to a missing recaptcha token. It started at 2024-11-04 09:37 UTC and was resolved at 2024-11-04 12:02 UTC, lasting 2 h 24 min. The vendor made environment changes to mitigate the issue and confirmed normal service.

GustoSome customers are experiencing slowness and errors in their Gusto dashboard

Gusto reported a major incident causing dashboard slowness and errors for some customers, affecting Gusto.com website, Payroll, Benefits, and HR. It started at 2024-10-31 17:09 UTC and was resolved at 2024-10-31 18:20 UTC, lasting 1 h 10 min; Gusto reported customers should no longer experience symptoms at 17:17 UTC.

QuickBooks OnlineError making a Full Payment Submission (FPS) in QuickBooks Core Payroll

Some QuickBooks Core Payroll customers in EMEA, QuickBooks Online - United Kingdom experienced Full Payment Submission (FPS) errors in a major-impact incident starting 2024-10-31 10:18 UTC. It lasted 34 min and was resolved at 2024-10-31 10:53 UTC, restoring FPS submissions; all other Core Payroll functionality remained operational.

PPandaDocEmail delivery issues

PandaDoc reported a major email-delivery incident affecting EU and US & Global document sending/opening from 2024-10-30 12:23 UTC to 2024-10-31 20:49 UTC, lasting 32 h 26 min. Fraudulent signups triggered Spamhaus phishing flags, blocklisting an email domain and IP ranges; some emails went to spam or quarantine. PandaDoc implemented mitigation measures, suggested link/SMS alternatives, and marked the incident resolved.

MezmoPipeline Web UI is Unresponsive

Mezmo’s critical Pipeline Web UI outage started 2024-10-26 01:00 UTC and resolved 2024-10-26 20:23 UTC (19 h 23 min). Pages failed to load, then loaded slowly with metrics sometimes unavailable; ingress and egress were unaffected. The postmortem limited customer impact to the first four hours; no cause or specific fix was stated.

PPendoKnown issue affecting Pendo for Employees (P4E) customers using Microsoft Edge (V130)

A major issue affected Pendo for Employees (P4E) customers on Microsoft Windows: Edge V130 prevented extensions using managed storage from functioning. The incident records a start and resolution at 2024-10-25 16:02 UTC (computed duration: 0 min). Pendo confirmed Microsoft's fix in Edge V130.0.2849.56; affected users must update to that release.

CCourierDelayed Message Processing

CCloseIssues syncing with Calendly

Close reported a major incident affecting calendar event syncing through its Calendly integration, starting at 2024-10-24 20:17 UTC and resolved at 2024-10-24 21:00 UTC. The incident lasted 42 min; no cause or fix was stated.

DDremioEngine scale up failing in Dremio Cloud (EU)

Dremio reported a major incident preventing Azure engines using PrivateLink connectivity to Dremio Cloud (EU) from scaling up. It started at 2024-10-24 08:44 UTC and was resolved at 2024-10-24 09:36 UTC, lasting 51 min. Dremio mitigated the issue and confirmed successful engine scale-ups in customer environments.

MMailjetEvent sending issues related to Event API

AAptibleLong load balancer registration times

RRudderStackIngestion got affected for customers on pro tier in EU region.

RudderStack reported a major ingestion outage for pro tier customers in the EU region: JS/mobile SDK events would retry, while server-side SDK events risked data loss. Listed start: 2024-10-22 10:46 UTC; listed resolution: 2024-10-22 10:00 UTC; computed duration: unknown. The resolved update reports impact from 10:00AM to 10:15AM UTC.

ContentstackContentstack - AWS EU - Publishing and Web Application issue

OOmni AnalyticsModel and Query Loading Failures in East USA

Omni Analytics reported a critical East USA outage affecting model and query loading: reports and new analyses loaded slowly or failed. The incident ran from 2024-10-18 15:08 UTC to 2024-10-18 20:50 UTC, lasting 5 h 41 min. The issue was isolated and mitigated by 16:02 UTC, and the service subsequently remained stable.

OOmni AnalyticsModel and Query Loading Failures in East USA

Omni Analytics reported a critical East USA incident: reports and new analyses loaded slowly or failed to load from 2024-10-17 20:46 UTC until 2024-10-17 23:43 UTC, lasting 2 h 57 min. A fix was applied at 10-17 22:19 UTC; the vendor said it would perform a root cause analysis and take steps to prevent recurrence.

GustoAll customers are unable to use our chat option for support.

Gusto’s Chat System outage prevented all customers from using support chat, with critical impact. It started at 2024-10-17 14:35 UTC and was resolved at 2024-10-17 16:08 UTC, lasting 1 h 32 min. Gusto implemented a fix and directed customers to request a call through their account’s Support tab during the outage.

SingleStoreCapacity issues in Azure Southcentral

PPipedreamExperiencing significant delay with HTTP event sources

Pipedream reported a major degradation causing significant delays with HTTP event sources, starting 2024-10-16 17:53 UTC and resolved at 2024-10-16 18:43 UTC (49 min). The backend service powering HTTP events encountered an issue during maintenance failover; services returned online and event processing caught up.

ngrokNew endpoints failing to start in all regions

IIterableIssues impacting Email Sending

Iterable reported a major incident affecting Global Campaign Sends and Global Proof Sends, with email send errors and campaign send skips linked to a third-party ESP’s API. It started at 2024-10-15 16:06 UTC and resolved at 2024-10-15 21:32 UTC, lasting 5 h 25 min. The vendor released a fix, and all affected messages were successfully processed.

DDropbox SignWe are currently experiencing performance problems with Forms

Dropbox Sign reported major performance problems affecting form sending and form filling through Dropbox Forms, starting at 2024-10-14 18:26 UTC. The incident was resolved at 2024-10-14 18:38 UTC after 12 min; no cause or specific fix was stated.

IimgixUnable to connect to dashboard

CCourierCourier Inbox FetchMessages Connectivity Issues