What vendors changed

GustoSome customers are experiencing an error when accessing Gusto.

Some customers encountered errors accessing Gusto, affecting the API and Gusto.com website. Gusto classified the incident as major; it ran from 2024-09-17 01:31 UTC to 2024-09-17 04:50 UTC, lasting 3 h 19 min. A fix was implemented at 2024-09-17 03:05 UTC, followed by monitoring and resolution.

LoopsPrimary database outage

FFrontDelays in sending and receiving Gmail messages [gmail]

Front reported a major incident delaying Gmail message sending and receiving, initially in us-west-2 and later across multiple regions. It started at 2024-09-13 15:38 UTC and resolved at 2024-09-13 19:25 UTC, lasting 3 h 47 min. Front suspected Gmail rate limiting, drained the message backlog, and reported all channels back to normal.

CCloseApplication Loading Issue

Close reported a major incident affecting Application UI: intermittent loading issues began at 2024-09-12 19:29 UTC and were resolved at 2024-09-12 21:01 UTC, lasting 1 h 31 min. Its postmortem states that the app and API were severely degraded by a partial outage of its back end MongoDB database.

QuickBooks OnlineQBO Switch Company Issue

QuickBooks Online reported a major outage: some customers could not use Switch Company because the option was missing under the Gear Icon. It started at 2024-09-12 14:51 UTC and was resolved at 2024-09-12 16:09 UTC, lasting 1 h 18 min; a fix was implemented, and logging out and back in was offered as a workaround.

Opsgenie, Trelloand BitbucketUsers are experiencing reCaptcha errors while signing up

Opsgenie reported a critical incident preventing successful signup due to reCaptcha errors, affecting EU - Signup, Login & Authorization and Signup, Login & Authorization. It started at 2024-09-12 02:08 UTC and was resolved at 2024-09-12 03:28 UTC, lasting 1 h 20 min; Opsgenie said it identified the root cause but provided no details.

CCensusPartial Sync Outage

Census reported a major Sync Engine outage starting 2024-09-11 19:00 UTC, lasting 5 h 5 min and resolved at 2024-09-12 00:06 UTC. Empty unload batches could fail with file or Parquet conversion errors; after a fix, some syncs encountered function errors, primarily advanced sync engine mirror syncs with no deletes.

GustoSome Users are unable to access Gusto

Some users could not access Gusto in a major-impact outage affecting the Gusto.com website, Payroll, Benefits and HR. It started at 2024-09-10 20:57 UTC and lasted 37 min, resolving at 2024-09-10 21:34 UTC; a fix was implemented at 21:05 UTC, though some pages were still loading slowly at 21:17 UTC.

NNordLayerConnectivity issues with VPN servers in New York, USA

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in New York, USA, impacting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It lasted 3 h 21 min, from 2024-09-10 17:44 UTC to 2024-09-10 21:05 UTC; the service provider implemented a fix, and NordLayer reported no further connectivity issues.

FigmaService disruption

Figma reported a major disruption affecting the Real-time collaboration server, with intermittent errors that could cause file access permission errors. It started at 2024-09-10 07:57 UTC and was resolved at 2024-09-10 11:24 UTC, lasting 3 h 26 min. Figma reported implementing a fix at 09-10 10:17 UTC.

PPendoPendo Resource Center Missing

Pendo's Resource Center was missing from its own application, affecting Pendo UI in an incident rated major. It started at 2024-09-10 02:17 UTC and was resolved at 2024-09-10 02:42 UTC, lasting 24 min; no cause or fix was stated.

ContentstackContentstack - GCP NA - Content Management API and Web Application Outage

PusherWe are experiencing a major outage in our US2 cluster

Pusher’s critical US2 outage affected most traffic, including Channels presence channels, REST API, Webhooks and WebSocket client API. It started at 2024-09-08 18:00 UTC and was resolved at 2024-09-08 23:06 UTC, lasting 5 h 5 min. Pusher reported deploying a fix and restoring services; no cause was specified.

BitbucketBitbucket Cloud website performance degradation

Bitbucket Cloud website users experienced slower-than-expected response times in an incident rated critical by the vendor. It started at 2024-09-06 06:49 UTC and was resolved at 2024-09-06 07:05 UTC, lasting 16 min; a fix was implemented, but no cause was stated.

RRudderStackIngestion failure for a subset of webhook source events

RudderStack reported a major ingestion failure affecting a subset of webhook source events in EU and US Data Ingestion. The incident started at 2024-09-05 15:00 UTC and was resolved at 2024-09-05 15:30 UTC, lasting 30 min; RudderStack identified the issue and reported implementing a fix, without specifying the cause.

BitbucketBitbucket Webhooks and Pipelines on push and pull request not being triggered

Bitbucket reported a major outage: Connect Webhooks were not delivered, and Pipelines on push and pull request were not triggered; manual and scheduled pipelines still worked. It started at 2024-09-05 02:19 UTC and was resolved at 2024-09-05 04:23 UTC, lasting 2 h 3 min. A fix was implemented and monitored before resolution.

SSigma ComputingSnowflake tables not loading on AWS US West 2

Sigma Computing reported a major degradation affecting AWS US West 2: long-running Snowflake table queries in Sigma Workbooks could time out. Updates placed onset around 14:00 UTC on September 4, 2024; the incident record started at 14:57 UTC, with a computed duration of 52 min. Temporary mitigation proved effective; resolution was recorded at 15:50 UTC.

Jupyter (Anaconda)Errors for sign-in to anaconda.cloud

Anaconda reported a major incident causing HTTP 503 errors when signing in to anaconda.cloud, affecting AI Platform Application - anaconda.com/app and Anaconda.cloud - Anaconda.cloud Frontend Website. It started at 2024-08-30 14:32 UTC and was resolved at 2024-08-30 15:56 UTC, lasting 1 h 23 min; Anaconda said it identified the root cause and deployed a fix.

CCriblUsers unable to login to cloud portal

PostmanWorkspaces are not loading

Postman workspaces failed to load in a major incident affecting browser and desktop platforms, including Europe and US components. It started at 2024-08-26 10:08 UTC and was resolved at 2024-08-26 12:20 UTC, lasting 2 h 11 min. Postman implemented a fix but did not state a cause.

CCensusSync execution halted

Census’s critical Sync Engine outage paused all production syncs starting 2024-08-23 17:41 UTC; many syncs failed after partial restoration. Census ruled out duplicate-record concerns and deployed a fix, restoring expected success rates with backlog delays. The incident was resolved at 2024-08-23 19:15 UTC after 1 h 33 min.

SubstackSubstack increased latency and error rate

Substack reported a critical incident with increased latency and errors affecting Substack.com, Substack publications, Writer dashboards, and the Substack app. It started at 2024-08-23 17:36 UTC and was resolved at 2024-08-23 19:06 UTC, lasting 1 h 29 min. Substack said it identified the cause but did not disclose it.

UpstashQStash API Not Reachable

Upstash’s QStash API was unreachable in an incident affecting QStash and EU-CENTRAL-1, rated critical by the vendor. It started at 2024-08-23 05:32 UTC and was resolved at 2024-08-23 06:01 UTC, lasting 28 min; Upstash reported identifying the issue and implementing a fix but provided no cause or fix details.

PPipedreamEvent History inaccessible

Pipedream reported a major outage of Event History starting 2024-08-22 21:16 UTC, leaving the event history UI inaccessible because its backend service was down. The incident lasted 3 h 20 min and was resolved at 2024-08-23 00:37 UTC, when the UI was restored and events had caught up processing.

SSecureframeOffice365: Some user logins and some integration syncs failing

Secureframe reported a major Web Application incident affecting some Office365 user logins and integration syncs, starting 2024-08-20 15:00 UTC and resolved at 2024-08-20 16:25 UTC; the recorded duration was 1 h 24 min. A fix was implemented at 08-20 16:15 UTC, but no cause was stated.

CCourierDelay in Message Processing

TypeformTypeform Admin & Submissions - Can't log in & unable to submit forms

Typeform reported a critical outage affecting admin login and form submissions starting at 2024-08-19 13:08 UTC. The incident lasted 16 min and was resolved at 2024-08-19 13:24 UTC after a fix was implemented; no cause was stated.

MailgunIntermittent errors for login.mailgun.com and app.mailgun.com

Mailgun reported a major incident causing intermittent errors at login.mailgun.com and app.mailgun.com, affecting Control Panel and Customer Tools - Control Panel. It started at 2024-08-17 17:35 UTC and lasted 22 min, resolving at 2024-08-17 17:57 UTC; a fix was implemented at 17:52 UTC.

CCensusLinkedin Syncs Failing

Census reported a major Sync Engine outage: Linkedin syncs failed because its API version had been deprecated. The incident ran from 2024-08-16 16:57 UTC to 2024-08-16 18:36 UTC, lasting 1 h 39 min. Census fixed the issue by updating the API version and said affected syncs should succeed on their next run.

HHexKernel Unavailability

Hex reported a critical incident affecting Main site and Single tenant stacks: some users could not acquire kernels or run projects. It started at 2024-08-15 18:32 UTC and lasted 3 h 5 min, resolving at 2024-08-15 21:37 UTC; Hex reported implementing a fix but did not state a cause.

CCriblSplunk Single Instance and LB Unexpectedly Closing Connections

PPipedreamDegraded functionality for git-synced projects

Pipedream reported a major GitHub Sync outage starting 2024-08-14 23:50 UTC, resolved at 2024-08-15 00:29 UTC after 39 min. GitHub syncing was unavailable due to a downstream outage; event viewing and monitoring remained available, and workflow execution was generally unaffected, though workflows relying on GitHub webhooks or APIs could be affected.

SubstackIncreased error rates

Substack reported critical increased error rates affecting Substack.com, Substack publications, and Writer dashboards, starting at 2024-08-13 19:24 UTC and resolved at 2024-08-13 19:42 UTC; the recorded duration was 17 min. Substack stated that a fix was implemented but did not specify the cause.

CockroachDBIncreased error rate in Serverless restore operations

CockroachDB reported a major incident involving increased restore errors in Serverless clusters, affecting CockroachDB Basic/Standard and CockroachDB Serverless. It started at 2024-08-09 21:37 UTC and was resolved at 2024-08-09 22:33 UTC, lasting 56 min. CockroachDB corrected a misconfiguration causing the errors and applied the fix to all serverless clusters.

AAblyIssue with Ably Queues in eu-west-1

Ably reported a major outage affecting Ably Queues and Ably Reactor Queues in Europe (Ireland), eu-west-1, starting 2024-08-09 13:40 UTC and resolved at 2024-08-09 13:44 UTC (4 min). An unresponsive coordinator node made https://eu-west-1-a-queue.ably.io:15671 unresponsive; Ably said it rectified the issue.

QuickBooks OnlineSome customers are unable to access Invoices page in QuickBooks Online.

QuickBooks Online reported a major outage affecting some customers’ access to the Invoices page in Canada and the United States, starting 2024-08-08 16:52 UTC and resolved at 2024-08-08 17:41 UTC (48 min). A fix was implemented; the resolution confirmed restored access to invoices, purchase orders and pay functions.

GustoWe are aware that some customers are experiencing issues running payroll.

Some Gusto customers experienced issues running payroll in an incident rated major, with Payroll, Benefits, and HR listed as affected components. It started at 2024-08-05 22:04 UTC and was resolved at 2024-08-05 22:14 UTC; the recorded duration was 9 min. Gusto did not state a cause or fix.

TypeformTypeform Sign Up - Registration is not available

Typeform reported a major registration outage affecting users trying to sign up, starting at 2024-08-05 13:20 UTC. The incident lasted 30 min and was resolved at 2024-08-05 13:51 UTC after a fix was implemented; no cause was stated.

SubstackDelayed push notifications due to OneSignal outage

Substack reported major delays to Push Notifications caused by an outage at its delivery provider, OneSignal. The incident started at 2024-08-04 19:08 UTC and was resolved at 2024-08-04 20:32 UTC, lasting 1 h 23 min.

DagsterDegraded Run Timeline Performance

Dagster reported major degradation of the Dagster Cloud UI run timeline due to overfetching runs, affecting timeline performance. The incident started at 2024-08-01 19:31 UTC and lasted 1 h 48 min, resolving at 2024-08-01 21:19 UTC; a fix was implemented at 08-01 21:08 UTC.

NNordLayerConnectivity issues with VPN servers in Montréal, Canada

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in Montréal, Canada, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It lasted 1 h 54 min, from 2024-08-01 15:00 UTC to 2024-08-01 16:54 UTC. The service provider implemented a fix, and NordLayer confirmed no further connectivity issues.

QuickBooks OnlineSome customers are unable to process credit card payments in QuickBooks Desktop.

Some QuickBooks Merchant Account customers in the United States could not process credit card payments in QuickBooks Desktop during a major outage from 2024-07-31 21:51 UTC to 2024-08-01 01:43 UTC, lasting 3 h 52 min. QuickBooks suggested manually processing urgent payments through the Merchant Service Center; payment processing was subsequently restored.

PPendoIncreased latency for data ingestion, guides, and other services

Pendo reported a major incident causing increased latency and error rates for data ingestion, guides, and other services, affecting analytics, guide delivery and targeting, NPS, and Resource Center. It started at 2024-07-31 20:30 UTC and was resolved at 2024-07-31 20:52 UTC, lasting 22 min; no cause or specific fix was stated.

MMailjetLogin issues with Mailjet Application

PyPI (Python Infrastructure)PyPI Backend Outage

MapboxAWS system-wide outage impacting Mapbox traffic and ETAsEvent

Mapbox reported a critical outage affecting live traffic, live incidents and ETAs through its API and Navigation - API, citing instability in AWS core services. It started at 2024-07-30 23:07 UTC and was resolved at 2024-07-31 12:17 UTC, lasting 13 h 9 min; Mapbox reported traffic services fully operational before resolution.

MapboxIncreased Vector Tile errorsEvent

Mapbox reported a major incident affecting the live traffic vector tile service, with increased error responses and service disruptions starting 2024-07-30 23:02 UTC. It lasted 7 h 38 min and was resolved at 2024-07-31 06:41 UTC after several fixes; Vector Tiles API error rates subsided and service was fully operational.

TypeformTypeform Follow-Ups - Unavailable - Follow-Up emails are not received.

Typeform reported a major Follow-Ups outage starting 2024-07-30 18:10 UTC, delaying Respondent and Self Follow-Up emails. Resolved at 2024-07-30 19:53 UTC after 1 h 42 min, the incident ended with a fix implemented, delayed messages delivered, and new Follow-Up messages delivering as expected.