What vendors changed

CCensusLinkedin Syncs Failing

Census reported a major Sync Engine outage: Linkedin syncs failed because its API version had been deprecated. The incident ran from 2024-08-16 16:57 UTC to 2024-08-16 18:36 UTC, lasting 1 h 39 min. Census fixed the issue by updating the API version and said affected syncs should succeed on their next run.

HHexKernel Unavailability

Hex reported a critical incident affecting Main site and Single tenant stacks: some users could not acquire kernels or run projects. It started at 2024-08-15 18:32 UTC and lasted 3 h 5 min, resolving at 2024-08-15 21:37 UTC; Hex reported implementing a fix but did not state a cause.

CCriblSplunk Single Instance and LB Unexpectedly Closing Connections

PPipedreamDegraded functionality for git-synced projects

Pipedream reported a major GitHub Sync outage starting 2024-08-14 23:50 UTC, resolved at 2024-08-15 00:29 UTC after 39 min. GitHub syncing was unavailable due to a downstream outage; event viewing and monitoring remained available, and workflow execution was generally unaffected, though workflows relying on GitHub webhooks or APIs could be affected.

SubstackIncreased error rates

Substack reported critical increased error rates affecting Substack.com, Substack publications, and Writer dashboards, starting at 2024-08-13 19:24 UTC and resolved at 2024-08-13 19:42 UTC; the recorded duration was 17 min. Substack stated that a fix was implemented but did not specify the cause.

CockroachDBIncreased error rate in Serverless restore operations

CockroachDB reported a major incident involving increased restore errors in Serverless clusters, affecting CockroachDB Basic/Standard and CockroachDB Serverless. It started at 2024-08-09 21:37 UTC and was resolved at 2024-08-09 22:33 UTC, lasting 56 min. CockroachDB corrected a misconfiguration causing the errors and applied the fix to all serverless clusters.

AAblyIssue with Ably Queues in eu-west-1

Ably reported a major outage affecting Ably Queues and Ably Reactor Queues in Europe (Ireland), eu-west-1, starting 2024-08-09 13:40 UTC and resolved at 2024-08-09 13:44 UTC (4 min). An unresponsive coordinator node made https://eu-west-1-a-queue.ably.io:15671 unresponsive; Ably said it rectified the issue.

QuickBooks OnlineSome customers are unable to access Invoices page in QuickBooks Online.

QuickBooks Online reported a major outage affecting some customers’ access to the Invoices page in Canada and the United States, starting 2024-08-08 16:52 UTC and resolved at 2024-08-08 17:41 UTC (48 min). A fix was implemented; the resolution confirmed restored access to invoices, purchase orders and pay functions.

GustoWe are aware that some customers are experiencing issues running payroll.

Some Gusto customers experienced issues running payroll in an incident rated major, with Payroll, Benefits, and HR listed as affected components. It started at 2024-08-05 22:04 UTC and was resolved at 2024-08-05 22:14 UTC; the recorded duration was 9 min. Gusto did not state a cause or fix.

TypeformTypeform Sign Up - Registration is not available

Typeform reported a major registration outage affecting users trying to sign up, starting at 2024-08-05 13:20 UTC. The incident lasted 30 min and was resolved at 2024-08-05 13:51 UTC after a fix was implemented; no cause was stated.

SubstackDelayed push notifications due to OneSignal outage

Substack reported major delays to Push Notifications caused by an outage at its delivery provider, OneSignal. The incident started at 2024-08-04 19:08 UTC and was resolved at 2024-08-04 20:32 UTC, lasting 1 h 23 min.

DagsterDegraded Run Timeline Performance

Dagster reported major degradation of the Dagster Cloud UI run timeline due to overfetching runs, affecting timeline performance. The incident started at 2024-08-01 19:31 UTC and lasted 1 h 48 min, resolving at 2024-08-01 21:19 UTC; a fix was implemented at 08-01 21:08 UTC.

NNordLayerConnectivity issues with VPN servers in Montréal, Canada

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in Montréal, Canada, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It lasted 1 h 54 min, from 2024-08-01 15:00 UTC to 2024-08-01 16:54 UTC. The service provider implemented a fix, and NordLayer confirmed no further connectivity issues.

QuickBooks OnlineSome customers are unable to process credit card payments in QuickBooks Desktop.

Some QuickBooks Merchant Account customers in the United States could not process credit card payments in QuickBooks Desktop during a major outage from 2024-07-31 21:51 UTC to 2024-08-01 01:43 UTC, lasting 3 h 52 min. QuickBooks suggested manually processing urgent payments through the Merchant Service Center; payment processing was subsequently restored.

PPendoIncreased latency for data ingestion, guides, and other services

Pendo reported a major incident causing increased latency and error rates for data ingestion, guides, and other services, affecting analytics, guide delivery and targeting, NPS, and Resource Center. It started at 2024-07-31 20:30 UTC and was resolved at 2024-07-31 20:52 UTC, lasting 22 min; no cause or specific fix was stated.

MMailjetLogin issues with Mailjet Application

PyPI (Python Infrastructure)PyPI Backend Outage

MapboxAWS system-wide outage impacting Mapbox traffic and ETAsEvent

Mapbox reported a critical outage affecting live traffic, live incidents and ETAs through its API and Navigation - API, citing instability in AWS core services. It started at 2024-07-30 23:07 UTC and was resolved at 2024-07-31 12:17 UTC, lasting 13 h 9 min; Mapbox reported traffic services fully operational before resolution.

MapboxIncreased Vector Tile errorsEvent

Mapbox reported a major incident affecting the live traffic vector tile service, with increased error responses and service disruptions starting 2024-07-30 23:02 UTC. It lasted 7 h 38 min and was resolved at 2024-07-31 06:41 UTC after several fixes; Vector Tiles API error rates subsided and service was fully operational.

TypeformTypeform Follow-Ups - Unavailable - Follow-Up emails are not received.

Typeform reported a major Follow-Ups outage starting 2024-07-30 18:10 UTC, delaying Respondent and Self Follow-Up emails. Resolved at 2024-07-30 19:53 UTC after 1 h 42 min, the incident ended with a fix implemented, delayed messages delivered, and new Follow-Up messages delivering as expected.

SSpeechmaticsBatch SaaS and Portal - Partial Outage

MMailjetLogin issues with Mailjet Application

SSpeechmaticsRT SaaS - Partial Outage

TrelloIssue with search and moving cards for some users

Some Trello.com customers experienced intermittent errors moving cards between boards or searching for cards. Trello rated the outage major; it started at 2024-07-25 19:31 UTC and was resolved at 2024-07-25 20:27 UTC, lasting 56 min. Trello confirmed normal operation but did not specify the cause or fix.

PulumiPulumi Copilot outage

Pulumi Copilot experienced a partial outage with major impact due to connectivity issues with Microsoft Azure. It started at 2024-07-22 23:57 UTC and was resolved at 2024-07-23 00:30 UTC, lasting 33 min.

FigmaService disruption

Figma reported a major partial availability disruption affecting APIs & Web Application, AWS Infrastructure, Real-time collaboration server, and REST API. The incident record started at 2024-07-22 13:58 UTC and was resolved at 2024-07-22 14:58 UTC (computed duration: 59 min); Figma reported actual availability issues from 13:38 to 13:48 UTC and said it had addressed the root cause.

CockroachDBCockroachDB Unavailable in Azure Central US

CockroachDB reported a major outage affecting CockroachDB Advanced and CockroachDB Dedicated clusters with nodes in Azure Central US, which may have been unavailable starting 2024-07-18 23:20 UTC. The incident lasted 15 h 49 min and was resolved at 2024-07-19 15:09 UTC, after CockroachDB verified all Azure clusters were operational and backup jobs had completed.

CockroachDBDedicated Cluster Operations Errors

CockroachDB reported a major outage preventing Dedicated cluster creation and edits and potentially affecting Cloud Console pages connecting to Serverless and Dedicated clusters; SQL connections worked normally. Starting 2024-07-18 01:56 UTC, it lasted 12 h 49 min and was resolved at 2024-07-18 14:45 UTC after the vendor addressed an unspecified underlying cause.

CCriblSearches terminated prematurely

AirtableAirtable unavailable for some users

Airtable reported a major outage preventing some users from accessing Airtable, starting at 2024-07-15 23:34 UTC and resolved at 2024-07-16 00:49 UTC, with a reported duration of 1 h 14 min. A fix was implemented by 07-16 00:08 UTC; Airtable confirmed services were functioning normally at resolution.

CodecovUploads are failing

Uploads were failing on Codecov Backend in an incident rated critical, starting at 2024-07-15 13:45 UTC and resolved at 2024-07-15 13:56 UTC after 11 min. Codecov did not state a cause or fix.

RRudderStackIntercom destination got affected for all customer tiers.

RudderStack reported a major Intercom destination outage affecting all customer tiers across EU and US data processing pipelines. Intercom’s WAF blocked requests with the User-Agent header set to RudderLabs. The incident started at 2024-07-15 12:23 UTC and was resolved at 2024-07-15 13:54 UTC, lasting 1 h 31 min; RudderStack reported the issue fixed.

SSpeechmaticsBatch transcription will speech capabilities impacted

LLightning AIStudios not starting

AAblyPlanned maintenance for ably.com and control.ably.net

Ably reported major-impact planned maintenance that made ably.com and control.ably.net unavailable; the realtime system was unaffected. The recorded start is 2024-07-09 10:23 UTC, but resolution is dated 2024-03-19 08:00 UTC; computed duration is unknown, while Ably’s update states maintenance lasted 30 minutes.

CCourierRendering Errors for Email Templates

PostmanPostman Login is impacted

Postman reported a major incident affecting Postman Login and US - Postman Login, starting at 2024-07-04 03:03 UTC and resolved at 2024-07-04 03:57 UTC; the computed duration was 53 min. A fix was implemented at 03:43 UTC, but no cause or fix details were provided.

OpsgenieSome products are hard down

Opsgenie reported a major outage affecting alert and incident flows, notification delivery in EU and US, and its public website. The incident ran from 2024-07-03 20:51 UTC to 2024-07-04 00:36 UTC (3 h 45 min); reported downtime was between 03-07-2024 20:08 UTC and 03-07-2024 20:31 UTC. Opsgenie mitigated the problem and restored service.

Jupyter (Anaconda)Blank pages for anaconda.cloud

Some users saw blank pages when rejecting cookies, affecting AI Platform Application - anaconda.com/app and Anaconda.cloud - Anaconda.cloud Frontend Website. Jupyter (Anaconda) rated the incident major; it ran from 2024-07-03 13:15 UTC to 2024-07-03 14:29 UTC, lasting 1 h 14 min. The vendor marked it resolved and said monitoring would continue.

JFrogJFrog Xray services intermittent outage affecting a subset of customers in AWS US East and EUC Regions

JFrog Xray experienced a major intermittent outage affecting a subset of customers in AWS US East1 (N. Virginia) and Europe - Central1 (Frankfurt). It started at 2024-07-03 12:25 UTC and was resolved at 2024-07-03 13:33 UTC, lasting 1 h 8 min; JFrog reported implementing a fix.

AsanaAsana unreachable for some users

Asana reported a major outage starting 2024-07-01 22:30 UTC that prevented some users from connecting; the recorded duration is 0 min. Its postmortem attributes the outage to an invalid configuration affecting approximately 18% of customers in its US datacenter, with resolution delayed by an unrelated infrastructure configuration management tooling issue. Asana said the issue was fixed.

CCourierAWS serviceUnavailable Outage

TypeformTypeform Platform - Users are unable to load forms

Typeform reported a major outage affecting users opening or loading forms, starting 2024-06-25 11:45 UTC and resolved at 2024-06-25 12:23 UTC, lasting 37 min. A fix was implemented; Typeform said it would investigate the cause and prevention.

QuickBooks OnlineSome QuickBooks Online customers are unable to login

Some QuickBooks Online customers could not log in during a major-impact incident listing QuickBooks Online and QuickBooks Self-Employed components in Canada, the United Kingdom and the United States. It started at 2024-06-24 13:10 UTC and was resolved at 2024-06-24 14:04 UTC, lasting 54 min; no cause or specific fix was stated.

CCensusSync runs failing

A subset of Census Sync Engine runs failed in a major-impact incident starting 2024-06-22 13:29 UTC and lasting 14 min, resolved at 2024-06-22 13:43 UTC. Census identified a storage issue affecting a core database and applied a fix, after which error rates decreased.

LLogRocketIngestion Instability

LogRocket reported a major outage affecting Session Collection ingestion endpoints from 2024-06-21 15:00 UTC to 2024-06-21 18:22 UTC, lasting 3 h 22 min. Ingestion stabilized while a backlog remained; the backlog had fully recovered by resolution.

FFireboltIncident Impacting Engine Start and Ingest Process

Firebolt reported a major outage preventing engines from starting and impacting ingestion, from 2024-06-21 06:42 UTC to 2024-06-21 09:36 UTC (2 h 54 min). Engineering identified and resolved the issue, but no cause or specific fix was stated.

TrelloIntermittent error accessing content

Trello.com experienced a major incident with intermittent content-access errors affecting some Atlassian Cloud customers, starting 2024-06-21 00:18 UTC and lasting 2 h 29 min, until resolution at 2024-06-21 02:47 UTC. The vendor said it identified the root cause, mitigated the problem, and restored normal service.

PulumiPulumi Deployments Outage

Pulumi reported a major Deployments outage starting 2024-06-20 22:33 UTC: deployments using DockerHub-hosted images, including the default pulumi/pulumi image, failed due to a DockerHub outage. The incident was resolved at 2024-06-20 22:42 UTC, with a reported duration of 8 min.

HHexIntermittent 502s when accessing Hex

Hex reported major intermittent 502 errors affecting access to Main site and Single tenant stacks, starting 2024-06-20 19:18 UTC and resolved at 2024-06-20 20:32 UTC after 1 h 14 min. Hex said it was following a Cloudflare incident involving intermittent connectivity issues; no specific fix was stated.