What vendors changed

PPendoPendo login issue

Pendo reported a critical infrastructure issue affecting application login in Pendo UI, starting 2024-03-31 19:22 UTC and resolved at 2024-03-31 23:21 UTC, lasting 3 h 58 min. Fixes were reported at 03-31 20:39 UTC and 03-31 22:47 UTC, with renewed investigation between them.

JFrogJFrog Cloud Platform Unavailable For Some Customers on AWS US East

JFrog reported a major outage affecting some customers on AWS US East, with Artifactory, Distribution, Mission Control, Pipelines and Xray listed as affected. It started at 2024-03-28 16:38 UTC and was resolved at 2024-03-28 21:54 UTC, lasting 5 h 16 min; JFrog reported implementing a fix and confirmed all services operational.

JFrogJFrog Cloud Platform UI is Unavailable For Some Customers

JFrog reported a major Cloud Platform UI outage affecting some customers, with Artifactory in US - East1 (N. Virginia) - AWS listed as affected. It started at 2024-03-27 14:41 UTC and resolved at 2024-03-27 16:43 UTC, lasting 2 h 1 min. JFrog said it identified and resolved a frontend application limitation.

CCensusUI slowdown across several areas

Census reported a major slowdown in its Sync Management UI affecting Model Previews, Metadata Refreshes and Sync Alerts, starting 2024-03-26 21:53 UTC and resolved at 2024-03-26 22:47 UTC (53 min). Census attributed it to a faulty deploy and restored task processing by deploying the previous job processor version; metrics showed no impact on sync performance or scheduling.

CodaSite-wide outage

Coda reported a major site-wide outage affecting access to API, Doc List, Docs, Packs, and Published Docs. It started at 2024-03-26 19:40 UTC and was resolved at 2024-03-26 19:51 UTC, lasting 11 min; no cause or specific fix was stated.

UUnbounceNew Domains Cannot Be Set-up

Unbounce reported a major outage preventing new domain setup due to a third-party provider issue, starting 2024-03-26 16:38 UTC and resolved 2024-03-27 15:48 UTC (23 h 10 min). Affected setups retried automatically; at resolution, Unbounce said all domain functionality had been working for over 20 hours despite continuing provider issues.

TypeformTypeform Platform unavailable

Typeform reported a critical platform outage affecting administration, opening forms and submitting responses, starting 2024-03-24 09:51 UTC and resolved at 2024-03-24 11:17 UTC (1 h 25 min). An admin-platform fix was applied first, followed by a fix across all services; the root cause was identified but not disclosed.

Ccrates.iocrates.io unreachable

Ccrates.iocrates.io unavailable

AirtableSome users unable to connect to Airtable

Airtable reported a major outage preventing some users from connecting and leaving some bases unavailable; Automations also stopped running. It started at 2024-03-21 16:42 UTC and was resolved at 2024-03-22 00:27 UTC, lasting 7 h 44 min. Airtable implemented mitigations and fixes, fully re-enabled Automations, and confirmed services were functioning normally.

PulumiPulumi is down

Pulumi reported a major outage affecting API, Console, and Deployments starting 2024-03-20 00:43 UTC. It was resolved at 2024-03-20 00:47 UTC, with a computed duration of 3 min; no cause or specific fix was stated.

GustoWe are currently experiencing higher than normal call volume. During this time, please leverage help center for support. Thank you for your patience.

Gusto reported higher-than-normal call volume affecting its Phone System, rated critical, and directed customers to the help center for support. The incident started at 2024-03-19 21:13 UTC and was resolved at 2024-03-19 21:55 UTC, lasting 42 min; no cause or fix was stated.

UUnbounceEmail Notification Deliverability Issues

Unbounce’s major email deliverability incident affected Lead Notifications, Page Upload and Downloads, and User Invitations from 2024-03-18 21:14 UTC to 2024-04-03 23:35 UTC, lasting 16 days 2 h. Unbounce fixed an underlying issue causing spam/blocklisting and mail-server rejection; deliverability returned to pre-incident rates, but customers may still need to whitelist senders to avoid spam filtering.

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported a major incident affecting dedicated VPN server connectivity in South Africa, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2024-03-18 09:45 UTC and was resolved at 2024-03-18 14:22 UTC, lasting 4 h 37 min. The service provider implemented a fix, and NordLayer detected no further connectivity issues.

TypeformTypeform Checkout & Create Page - Unavailable

Typeform reported a critical outage of its Checkout & Create Page affecting Admin - Subscription and Subscription. It started at 2024-03-18 07:37 UTC and was resolved at 2024-03-18 07:47 UTC, lasting 9 min. Typeform identified a traffic interruption impacting checkout and implemented a fix; no cause was stated.

HHexKernels unavailable

Hex reported a critical outage affecting Kernels and Main site, with kernels unavailable starting at 2024-03-17 04:08 UTC. A fix was implemented by 2024-03-17 04:42 UTC, and the incident was resolved at 2024-03-17 04:50 UTC; the reported duration was 41 min.

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported a major outage involving connectivity issues with dedicated VPN servers in South Africa, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2024-03-15 08:30 UTC and was resolved at 2024-03-15 17:00 UTC, lasting 8 h 29 min. The service provider implemented a fix; no further connectivity issues were detected.

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported major connectivity issues with dedicated VPN servers in South Africa, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). The incident started at 2024-03-14 10:45 UTC and was resolved at 2024-03-14 15:43 UTC, lasting 4 h 57 min. The service provider implemented a fix, and no further connectivity issues were detected.

StatuspageStatuspage API is facing intermittent issues

Statuspage reported major intermittent API issues affecting Authenticated API, Hosted Pages - Public API, Management - Authenticated API, and Public API. The incident started at 2024-03-13 08:00 UTC and resolved at 2024-03-13 10:27 UTC, lasting 2 h 26 min. A fix was implemented and the API reported fully recovered at 03-13 08:47 UTC.

PyPI (Python Infrastructure)PyPI workers currently paused

HHexPlatform instability

Hex reported major platform instability affecting the Main site, starting at 2024-03-09 18:10 UTC and resolved at 2024-03-09 18:47 UTC after 37 min. Hex attributed the instability to a scheduled system change; no specific fix was stated.

GustoWe are aware of potential errors that customers may be experiencing with benefits features in Gusto. Our teams are working to resolve this as a top priority. Thank you for your patience at this time.

Gusto reported potential errors with benefits features in a critical-impact incident listing Payroll, Benefits, and HR as affected components. It started at 2024-03-09 00:48 UTC and was resolved at 2024-03-09 00:58 UTC; the computed duration was 9 min. No cause or specific fix was stated.

PPendoPossible issues with Pendo customers receiving Pendo emails

Pendo reported a major email outage: some users did not receive invitation/password reset emails, or Pendo emails were marked as Spam. It started 2024-03-07 20:59 UTC and was marked resolved 2024-03-12 13:22 UTC, lasting 4 days 16 h. Pendo advised customers’ IT teams to whitelist Pendo emails and linked sign-in troubleshooting guidance.

ngrok`jp` region is down for new tunnel starts and endpoint configuration changes

FFlagsmithIssues with flag updates

CodaOffline docs

Coda reported a major doc-connectivity outage from 2024-03-01 18:27 UTC to 2024-03-01 18:48 UTC, lasting 21 min. Docs showed as "Offline" but remained usable in offline mode, with edits saved when docs reconnected. The incident was resolved; no cause or specific fix was stated.

LLightdashPerformance degraded on dashboards and scheduled deliveries

Lightdash reported major performance degradation in EU and US Cloud dashboards and scheduled deliveries, with load times over one minute due to its feature flag provider’s outage. Starting 2024-02-28 16:34 UTC, the incident lasted 2 h 7 min and was resolved at 2024-02-28 18:42 UTC after a fix was deployed to all customers.

OpsgenieOpsgenie SAML login at eu region is not working

Opsgenie reported a major EU SAML login outage from 2024-02-27 14:25 UTC to 2024-02-27 14:58 UTC, lasting 33 min, due to a signature verification error in login certificates. Already logged-in customers were unaffected. Opsgenie deployed a fix and verified there were no further login issues.

DDremioUnable to start Azure-based project engines in Dremio Cloud

A major incident prevented Azure-based project engines from starting in Dremio Cloud - Europe and Dremio Cloud - US, affecting customers’ ability to start engines. It started at 2024-02-27 00:58 UTC and was resolved at 2024-02-27 08:30 UTC, lasting 7 h 31 min. Dremio rolled out a fix to all environments and verified engines were starting.

DagsterOutage in Dagster Cloud

Dagster Cloud’s API incident was classified major, started at 2024-02-26 17:44 UTC, and was resolved at 2024-02-26 18:05 UTC (20 min). Dagster reported an outage from 17:28 UTC - 17:39 UTC while rotating production secrets; traffic had resumed and recovery was monitored.

RRudderStackIngestion got affected for customers who are on enterprise.

RudderStack reported a critical outage affecting enterprise customers’ Data Ingestion (US), starting at 2024-02-23 23:33 UTC and lasting 1 h 59 min until resolution at 2024-02-24 01:32 UTC. A fix was rolled out at 02-24 00:52 UTC, followed by monitoring; no cause was stated.

TypeformTypeform Builder - Degraded - Builder not loading

Typeform reported a critical Builder loading incident that could affect form creation/editing, starting 2024-02-23 12:00 UTC and resolved at 2024-02-23 12:37 UTC (36 min). A fix was implemented; users still experiencing issues were advised to refresh or clear their browser cache.

SSigma ComputingOutage affecting Sigma organizations hosted on GCP

Sigma reported a critical outage affecting organizations hosted on GCP (US), preventing login and use of Sigma services. The incident record spans 2024-02-22 19:30 UTC to 2024-02-22 20:22 UTC (52 min), while updates report an approximately 18:00 UTC start and 18:35 UTC resolution. Sigma confirmed its mitigation measures were effective.

RRudderStackPublic API got affected partially post upgrade

RudderStack reported a major incident involving partial Public API disruption after an upgrade and a web application outage; ingestion and delivery remained normal. It started at 2024-02-21 05:12 UTC and was resolved at 2024-02-21 06:23 UTC, lasting 1 h 10 min.

SSecureframeSecurity Questionnaires Not Responding

Secureframe’s Security Questionnaires failed to load properly in a major outage starting 2024-02-19 17:36 UTC. The incident lasted 1 h 19 min and was resolved at 2024-02-19 18:56 UTC after a fix was shipped to production.

DDremioMultiple google services' outage affecting Dremio Cloud US control plane

Dremio Cloud US control plane availability was affected by a Google Cloud Platform multi-service outage in us-west1, rated major by Dremio. The incident started at 2024-02-14 19:43 UTC, lasted 3 h 43 min, and was resolved at 2024-02-14 23:27 UTC, when Dremio reported full recovery.

Ccrates.ioIntermittent timeouts on API endpoints

Ccrates.ioIntermittent timeouts on API endpoints

TrelloTrello is slow or unavailable

Trello reported a critical outage affecting API, Trello.com, and web and mobile apps, with slow or unavailable service starting 2024-02-13 09:30 UTC and resolving 2024-02-13 11:46 UTC (computed duration: 2 h 15 min). The postmortem cited long-running database queries slowing API responses; a fix was deployed and service restored.

ngrokOAuth, OIDC, and SAML modules for Edges degraded

KeeperAustralia data center vault login errors

Keeper reported critical vault login errors affecting Keeper Admin Console (AU) and Keeper Web Vault (AU), starting 2024-02-06 02:44 UTC and resolved at 2024-02-06 02:47 UTC (computed duration: 2 min). The vendor attributed API errors to an AWS ElastiCache error in AU (Sydney); restarting the affected resource cleared the errors.

WarpWarp Blocked By CrowdStrike

PostmanElevated error rates in multiple features

Postman reported a major incident affecting login and multiple workspace features across Postman Login, Postman Platform on Browser, and Postman Platform on Desktop. It started at 2024-02-05 11:49 UTC and was resolved at 2024-02-05 12:29 UTC, lasting 39 min; Postman reported rolling out an immediate fix and monitoring stability.

PPipedreamPipedream API down, affecting https://pipedream.com

Pipedream reported a critical API outage affecting https://pipedream.com and the REST API, starting at 2024-02-03 22:42 UTC. The incident lasted 4 min and was resolved at 2024-02-03 22:47 UTC; no cause or fix was stated.

Jupyter (Anaconda)community.anaconda.cloud is not loading correctly

community.anaconda.cloud (Discourse) was not loading correctly in a major outage that started at 2024-01-31 17:13 UTC and was resolved at 2024-01-31 20:49 UTC, lasting 3 h 35 min. Anaconda investigated the failure and engaged with the vendor; no cause or specific fix was stated.

SSecureframeEmployee onboarding redirect issues

Secureframe reported a major Web Application incident preventing employee onboarding from loading for customers with policies enabled and trainings disabled. It started at 2024-01-29 16:35 UTC and lasted 3 h 12 min, resolving at 2024-01-29 19:47 UTC; a fix was implemented at 01-29 19:43 UTC.

LLightdashIssues with deploying and creating preview environments

Lightdash reported a major incident affecting EU and US Cloud, causing empty Explores during deployments or Preview environment creation. It ran from 2024-01-29 15:40 UTC to 2024-01-29 17:08 UTC (1 h 28 min). Lightdash identified release 0.956.4, reverted Cloud instances, released a fix, and recommended self-hosted users upgrade to 0.965.6 and browser users hard-refresh.

OpsgenieMajor outages in heartbeat services pinging via email and email integration in EU region

Opsgenie reported a major EU outage affecting heartbeat updates via email and incoming email integrations, rejected from 13:45 UTC due to faulty domain configurations; other heartbeat functionality and integrations remained operational. The incident ran from 2024-01-29 15:08 UTC to 2024-01-29 15:29 UTC (21 min), with service restored after changes were reverted.

MapboxAuthentication errors on Matrix, Optimization, and Directions APIs

Mapbox reported a major incident involving HTTP 401 authentication errors for some Matrix, Optimization, and Directions API users in some regions. The recorded start is 2024-01-25 23:10 UTC and resolution is 2024-01-25 12:30 UTC; these timestamps conflict, and duration is unknown. Mapbox said the issue was identified and resolved.

Jupyter (Anaconda)Anaconda.cloud login issues

Some users could not log in to anaconda.cloud during a major-impact incident starting 2024-01-25 18:18 UTC and resolved at 2024-01-25 21:44 UTC, lasting 3 h 25 min. The vendor completed a fix and confirmed all users should be able to log in.