What vendors changed

DagsterSome Dagster Cloud API Requests failing with an Internal Server Error

Some Dagster Cloud API requests and a small number of runs failed with Internal Server Error exceptions due to a storage-engine timeout. The vendor rated the incident major; it ran from 2024-04-11 17:02 UTC to 2024-04-11 21:25 UTC, lasting 4 h 23 min. A fix was implemented and the incident resolved.

KeeperLogin errors - Resolved

Keeper reported a critical login outage affecting Keeper Admin Console (US) and Keeper Web Vault (US), starting 2024-04-11 12:55 UTC and resolved at 2024-04-11 13:12 UTC (17 min). Keeper attributed it to an underlying AWS database error that triggered a successfully handled failover; root-cause investigation with AWS continued.

SSecureframeUK Agents failing to report

Secureframe’s UK Agents failed to report in a major incident affecting Secureframe Agent from 2024-04-10 18:48 UTC to 2024-04-10 20:05 UTC, lasting 1 h 17 min. A fix was implemented and monitored before the incident was resolved.

CCensusSyncs not running

Census reported a critical Sync Engine outage in which syncs were not running, starting at 2024-04-09 23:32 UTC and lasting 3 h 9 min until resolution at 2024-04-10 02:41 UTC. A fix was implemented by 04-09 23:50 UTC, followed by monitoring.

OOmni AnalyticsDegraded Performance

Omni Analytics reported major degraded performance in EastUsa, with most queries failing to complete. The incident ran from 2024-04-09 18:44 UTC to 2024-04-10 01:04 UTC (6 h 19 min). Queries against a specific database type exhausted memory, repeatedly crashing querying until cancelled; Omni identified a fix for release that day and later marked the incident resolved.

OOmni AnalyticsSlow Queries

Omni Analytics reported major query degradation in EastUsa from 2024-04-08 16:55 UTC to 2024-04-08 17:05 UTC, lasting 10 min. Queries were slow or delayed because an information-schema query spike after cache expiration triggered a thundering herd, compounded by missing autoscaling logic; the incident was resolved, with autoscaling improvements planned.

PPipedream502 Bad Gateway on pipedream.com

Pipedream reported a major outage affecting pipedream.com, with elevated CPU causing 502 errors. It started at 2024-04-05 23:26 UTC and was resolved at 2024-04-05 23:28 UTC, lasting 2 min.

SSecureframeSecureframe Agent unable to connect

Secureframe reported a critical outage: Secureframe Agent could not connect to internal services, leaving it unable to operate and/or report. It lasted 20 h 22 min, from 2024-04-04 21:20 UTC to 2024-04-05 17:42 UTC. Secureframe implemented a fix and said it was working on an Agent update to reduce susceptibility to similar interruptions.

HotjarIssues with billing system

Hotjar reported a critical billing-system outage starting 2024-04-02 15:02 UTC, with access to checkout and invoices temporarily disabled during investigation. The incident was resolved at 2024-04-02 17:41 UTC; the recorded duration was 2 h 38 min.

RootlyIntermittent Failures with User Sign In and Incident Creation

Rootly reported a major incident causing intermittent sign-in and incident-creation failures across Slack, Web, and API; Rootly On-Call remained unaffected. It started at 2024-04-01 14:53 UTC and resolved at 2024-04-01 15:19 UTC, lasting 25 min. Rootly reported implementing a fix but did not disclose the cause; an RCA was promised to customers.

PPendoPendo login issue

Pendo reported a critical infrastructure issue affecting application login in Pendo UI, starting 2024-03-31 19:22 UTC and resolved at 2024-03-31 23:21 UTC, lasting 3 h 58 min. Fixes were reported at 03-31 20:39 UTC and 03-31 22:47 UTC, with renewed investigation between them.

JFrogJFrog Cloud Platform Unavailable For Some Customers on AWS US East

JFrog reported a major outage affecting some customers on AWS US East, with Artifactory, Distribution, Mission Control, Pipelines and Xray listed as affected. It started at 2024-03-28 16:38 UTC and was resolved at 2024-03-28 21:54 UTC, lasting 5 h 16 min; JFrog reported implementing a fix and confirmed all services operational.

JFrogJFrog Cloud Platform UI is Unavailable For Some Customers

JFrog reported a major Cloud Platform UI outage affecting some customers, with Artifactory in US - East1 (N. Virginia) - AWS listed as affected. It started at 2024-03-27 14:41 UTC and resolved at 2024-03-27 16:43 UTC, lasting 2 h 1 min. JFrog said it identified and resolved a frontend application limitation.

CCensusUI slowdown across several areas

Census reported a major slowdown in its Sync Management UI affecting Model Previews, Metadata Refreshes and Sync Alerts, starting 2024-03-26 21:53 UTC and resolved at 2024-03-26 22:47 UTC (53 min). Census attributed it to a faulty deploy and restored task processing by deploying the previous job processor version; metrics showed no impact on sync performance or scheduling.

CodaSite-wide outage

Coda reported a major site-wide outage affecting access to API, Doc List, Docs, Packs, and Published Docs. It started at 2024-03-26 19:40 UTC and was resolved at 2024-03-26 19:51 UTC, lasting 11 min; no cause or specific fix was stated.

UUnbounceNew Domains Cannot Be Set-up

Unbounce reported a major outage preventing new domain setup due to a third-party provider issue, starting 2024-03-26 16:38 UTC and resolved 2024-03-27 15:48 UTC (23 h 10 min). Affected setups retried automatically; at resolution, Unbounce said all domain functionality had been working for over 20 hours despite continuing provider issues.

TypeformTypeform Platform unavailable

Typeform reported a critical platform outage affecting administration, opening forms and submitting responses, starting 2024-03-24 09:51 UTC and resolved at 2024-03-24 11:17 UTC (1 h 25 min). An admin-platform fix was applied first, followed by a fix across all services; the root cause was identified but not disclosed.

Ccrates.iocrates.io unreachable

Ccrates.iocrates.io unavailable

AirtableSome users unable to connect to Airtable

Airtable reported a major outage preventing some users from connecting and leaving some bases unavailable; Automations also stopped running. It started at 2024-03-21 16:42 UTC and was resolved at 2024-03-22 00:27 UTC, lasting 7 h 44 min. Airtable implemented mitigations and fixes, fully re-enabled Automations, and confirmed services were functioning normally.

PulumiPulumi is down

Pulumi reported a major outage affecting API, Console, and Deployments starting 2024-03-20 00:43 UTC. It was resolved at 2024-03-20 00:47 UTC, with a computed duration of 3 min; no cause or specific fix was stated.

GustoWe are currently experiencing higher than normal call volume. During this time, please leverage help center for support. Thank you for your patience.

Gusto reported higher-than-normal call volume affecting its Phone System, rated critical, and directed customers to the help center for support. The incident started at 2024-03-19 21:13 UTC and was resolved at 2024-03-19 21:55 UTC, lasting 42 min; no cause or fix was stated.

UUnbounceEmail Notification Deliverability Issues

Unbounce’s major email deliverability incident affected Lead Notifications, Page Upload and Downloads, and User Invitations from 2024-03-18 21:14 UTC to 2024-04-03 23:35 UTC, lasting 16 days 2 h. Unbounce fixed an underlying issue causing spam/blocklisting and mail-server rejection; deliverability returned to pre-incident rates, but customers may still need to whitelist senders to avoid spam filtering.

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported a major incident affecting dedicated VPN server connectivity in South Africa, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2024-03-18 09:45 UTC and was resolved at 2024-03-18 14:22 UTC, lasting 4 h 37 min. The service provider implemented a fix, and NordLayer detected no further connectivity issues.

TypeformTypeform Checkout & Create Page - Unavailable

Typeform reported a critical outage of its Checkout & Create Page affecting Admin - Subscription and Subscription. It started at 2024-03-18 07:37 UTC and was resolved at 2024-03-18 07:47 UTC, lasting 9 min. Typeform identified a traffic interruption impacting checkout and implemented a fix; no cause was stated.

HHexKernels unavailable

Hex reported a critical outage affecting Kernels and Main site, with kernels unavailable starting at 2024-03-17 04:08 UTC. A fix was implemented by 2024-03-17 04:42 UTC, and the incident was resolved at 2024-03-17 04:50 UTC; the reported duration was 41 min.

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported a major outage involving connectivity issues with dedicated VPN servers in South Africa, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2024-03-15 08:30 UTC and was resolved at 2024-03-15 17:00 UTC, lasting 8 h 29 min. The service provider implemented a fix; no further connectivity issues were detected.

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported major connectivity issues with dedicated VPN servers in South Africa, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). The incident started at 2024-03-14 10:45 UTC and was resolved at 2024-03-14 15:43 UTC, lasting 4 h 57 min. The service provider implemented a fix, and no further connectivity issues were detected.

StatuspageStatuspage API is facing intermittent issues

Statuspage reported major intermittent API issues affecting Authenticated API, Hosted Pages - Public API, Management - Authenticated API, and Public API. The incident started at 2024-03-13 08:00 UTC and resolved at 2024-03-13 10:27 UTC, lasting 2 h 26 min. A fix was implemented and the API reported fully recovered at 03-13 08:47 UTC.

PyPI (Python Infrastructure)PyPI workers currently paused

HHexPlatform instability

Hex reported major platform instability affecting the Main site, starting at 2024-03-09 18:10 UTC and resolved at 2024-03-09 18:47 UTC after 37 min. Hex attributed the instability to a scheduled system change; no specific fix was stated.

GustoWe are aware of potential errors that customers may be experiencing with benefits features in Gusto. Our teams are working to resolve this as a top priority. Thank you for your patience at this time.

Gusto reported potential errors with benefits features in a critical-impact incident listing Payroll, Benefits, and HR as affected components. It started at 2024-03-09 00:48 UTC and was resolved at 2024-03-09 00:58 UTC; the computed duration was 9 min. No cause or specific fix was stated.

PPendoPossible issues with Pendo customers receiving Pendo emails

Pendo reported a major email outage: some users did not receive invitation/password reset emails, or Pendo emails were marked as Spam. It started 2024-03-07 20:59 UTC and was marked resolved 2024-03-12 13:22 UTC, lasting 4 days 16 h. Pendo advised customers’ IT teams to whitelist Pendo emails and linked sign-in troubleshooting guidance.

ngrok`jp` region is down for new tunnel starts and endpoint configuration changes

FFlagsmithIssues with flag updates

CodaOffline docs

Coda reported a major doc-connectivity outage from 2024-03-01 18:27 UTC to 2024-03-01 18:48 UTC, lasting 21 min. Docs showed as "Offline" but remained usable in offline mode, with edits saved when docs reconnected. The incident was resolved; no cause or specific fix was stated.

LLightdashPerformance degraded on dashboards and scheduled deliveries

Lightdash reported major performance degradation in EU and US Cloud dashboards and scheduled deliveries, with load times over one minute due to its feature flag provider’s outage. Starting 2024-02-28 16:34 UTC, the incident lasted 2 h 7 min and was resolved at 2024-02-28 18:42 UTC after a fix was deployed to all customers.

OpsgenieOpsgenie SAML login at eu region is not working

Opsgenie reported a major EU SAML login outage from 2024-02-27 14:25 UTC to 2024-02-27 14:58 UTC, lasting 33 min, due to a signature verification error in login certificates. Already logged-in customers were unaffected. Opsgenie deployed a fix and verified there were no further login issues.

DDremioUnable to start Azure-based project engines in Dremio Cloud

A major incident prevented Azure-based project engines from starting in Dremio Cloud - Europe and Dremio Cloud - US, affecting customers’ ability to start engines. It started at 2024-02-27 00:58 UTC and was resolved at 2024-02-27 08:30 UTC, lasting 7 h 31 min. Dremio rolled out a fix to all environments and verified engines were starting.

DagsterOutage in Dagster Cloud

Dagster Cloud’s API incident was classified major, started at 2024-02-26 17:44 UTC, and was resolved at 2024-02-26 18:05 UTC (20 min). Dagster reported an outage from 17:28 UTC - 17:39 UTC while rotating production secrets; traffic had resumed and recovery was monitored.

RRudderStackIngestion got affected for customers who are on enterprise.

RudderStack reported a critical outage affecting enterprise customers’ Data Ingestion (US), starting at 2024-02-23 23:33 UTC and lasting 1 h 59 min until resolution at 2024-02-24 01:32 UTC. A fix was rolled out at 02-24 00:52 UTC, followed by monitoring; no cause was stated.

TypeformTypeform Builder - Degraded - Builder not loading

Typeform reported a critical Builder loading incident that could affect form creation/editing, starting 2024-02-23 12:00 UTC and resolved at 2024-02-23 12:37 UTC (36 min). A fix was implemented; users still experiencing issues were advised to refresh or clear their browser cache.

SSigma ComputingOutage affecting Sigma organizations hosted on GCP

Sigma reported a critical outage affecting organizations hosted on GCP (US), preventing login and use of Sigma services. The incident record spans 2024-02-22 19:30 UTC to 2024-02-22 20:22 UTC (52 min), while updates report an approximately 18:00 UTC start and 18:35 UTC resolution. Sigma confirmed its mitigation measures were effective.

RRudderStackPublic API got affected partially post upgrade

RudderStack reported a major incident involving partial Public API disruption after an upgrade and a web application outage; ingestion and delivery remained normal. It started at 2024-02-21 05:12 UTC and was resolved at 2024-02-21 06:23 UTC, lasting 1 h 10 min.

SSecureframeSecurity Questionnaires Not Responding

Secureframe’s Security Questionnaires failed to load properly in a major outage starting 2024-02-19 17:36 UTC. The incident lasted 1 h 19 min and was resolved at 2024-02-19 18:56 UTC after a fix was shipped to production.

DDremioMultiple google services' outage affecting Dremio Cloud US control plane

Dremio Cloud US control plane availability was affected by a Google Cloud Platform multi-service outage in us-west1, rated major by Dremio. The incident started at 2024-02-14 19:43 UTC, lasted 3 h 43 min, and was resolved at 2024-02-14 23:27 UTC, when Dremio reported full recovery.

Ccrates.ioIntermittent timeouts on API endpoints

Ccrates.ioIntermittent timeouts on API endpoints

TrelloTrello is slow or unavailable

Trello reported a critical outage affecting API, Trello.com, and web and mobile apps, with slow or unavailable service starting 2024-02-13 09:30 UTC and resolving 2024-02-13 11:46 UTC (computed duration: 2 h 15 min). The postmortem cited long-running database queries slowing API responses; a fix was deployed and service restored.

ngrokOAuth, OIDC, and SAML modules for Edges degraded