What vendors changed

NNordLayerConnectivity issues with VPN servers in South Africa

NordLayer reported major connectivity issues with dedicated VPN servers in South Africa, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). The incident started at 2024-03-14 10:45 UTC and was resolved at 2024-03-14 15:43 UTC, lasting 4 h 57 min. The service provider implemented a fix, and no further connectivity issues were detected.

StatuspageStatuspage API is facing intermittent issues

Statuspage reported major intermittent API issues affecting Authenticated API, Hosted Pages - Public API, Management - Authenticated API, and Public API. The incident started at 2024-03-13 08:00 UTC and resolved at 2024-03-13 10:27 UTC, lasting 2 h 26 min. A fix was implemented and the API reported fully recovered at 03-13 08:47 UTC.

PyPI (Python Infrastructure)PyPI workers currently paused

HHexPlatform instability

Hex reported major platform instability affecting the Main site, starting at 2024-03-09 18:10 UTC and resolved at 2024-03-09 18:47 UTC after 37 min. Hex attributed the instability to a scheduled system change; no specific fix was stated.

GustoWe are aware of potential errors that customers may be experiencing with benefits features in Gusto. Our teams are working to resolve this as a top priority. Thank you for your patience at this time.

Gusto reported potential errors with benefits features in a critical-impact incident listing Payroll, Benefits, and HR as affected components. It started at 2024-03-09 00:48 UTC and was resolved at 2024-03-09 00:58 UTC; the computed duration was 9 min. No cause or specific fix was stated.

PPendoPossible issues with Pendo customers receiving Pendo emails

Pendo reported a major email outage: some users did not receive invitation/password reset emails, or Pendo emails were marked as Spam. It started 2024-03-07 20:59 UTC and was marked resolved 2024-03-12 13:22 UTC, lasting 4 days 16 h. Pendo advised customers’ IT teams to whitelist Pendo emails and linked sign-in troubleshooting guidance.

ngrok`jp` region is down for new tunnel starts and endpoint configuration changes

FFlagsmithIssues with flag updates

CodaOffline docs

Coda reported a major doc-connectivity outage from 2024-03-01 18:27 UTC to 2024-03-01 18:48 UTC, lasting 21 min. Docs showed as "Offline" but remained usable in offline mode, with edits saved when docs reconnected. The incident was resolved; no cause or specific fix was stated.

LLightdashPerformance degraded on dashboards and scheduled deliveries

Lightdash reported major performance degradation in EU and US Cloud dashboards and scheduled deliveries, with load times over one minute due to its feature flag provider’s outage. Starting 2024-02-28 16:34 UTC, the incident lasted 2 h 7 min and was resolved at 2024-02-28 18:42 UTC after a fix was deployed to all customers.

OpsgenieOpsgenie SAML login at eu region is not working

Opsgenie reported a major EU SAML login outage from 2024-02-27 14:25 UTC to 2024-02-27 14:58 UTC, lasting 33 min, due to a signature verification error in login certificates. Already logged-in customers were unaffected. Opsgenie deployed a fix and verified there were no further login issues.

DDremioUnable to start Azure-based project engines in Dremio Cloud

A major incident prevented Azure-based project engines from starting in Dremio Cloud - Europe and Dremio Cloud - US, affecting customers’ ability to start engines. It started at 2024-02-27 00:58 UTC and was resolved at 2024-02-27 08:30 UTC, lasting 7 h 31 min. Dremio rolled out a fix to all environments and verified engines were starting.

DagsterOutage in Dagster Cloud

Dagster Cloud’s API incident was classified major, started at 2024-02-26 17:44 UTC, and was resolved at 2024-02-26 18:05 UTC (20 min). Dagster reported an outage from 17:28 UTC - 17:39 UTC while rotating production secrets; traffic had resumed and recovery was monitored.

RRudderStackIngestion got affected for customers who are on enterprise.

RudderStack reported a critical outage affecting enterprise customers’ Data Ingestion (US), starting at 2024-02-23 23:33 UTC and lasting 1 h 59 min until resolution at 2024-02-24 01:32 UTC. A fix was rolled out at 02-24 00:52 UTC, followed by monitoring; no cause was stated.

TypeformTypeform Builder - Degraded - Builder not loading

Typeform reported a critical Builder loading incident that could affect form creation/editing, starting 2024-02-23 12:00 UTC and resolved at 2024-02-23 12:37 UTC (36 min). A fix was implemented; users still experiencing issues were advised to refresh or clear their browser cache.

SSigma ComputingOutage affecting Sigma organizations hosted on GCP

Sigma reported a critical outage affecting organizations hosted on GCP (US), preventing login and use of Sigma services. The incident record spans 2024-02-22 19:30 UTC to 2024-02-22 20:22 UTC (52 min), while updates report an approximately 18:00 UTC start and 18:35 UTC resolution. Sigma confirmed its mitigation measures were effective.

RRudderStackPublic API got affected partially post upgrade

RudderStack reported a major incident involving partial Public API disruption after an upgrade and a web application outage; ingestion and delivery remained normal. It started at 2024-02-21 05:12 UTC and was resolved at 2024-02-21 06:23 UTC, lasting 1 h 10 min.

SSecureframeSecurity Questionnaires Not Responding

Secureframe’s Security Questionnaires failed to load properly in a major outage starting 2024-02-19 17:36 UTC. The incident lasted 1 h 19 min and was resolved at 2024-02-19 18:56 UTC after a fix was shipped to production.

DDremioMultiple google services' outage affecting Dremio Cloud US control plane

Dremio Cloud US control plane availability was affected by a Google Cloud Platform multi-service outage in us-west1, rated major by Dremio. The incident started at 2024-02-14 19:43 UTC, lasted 3 h 43 min, and was resolved at 2024-02-14 23:27 UTC, when Dremio reported full recovery.

Ccrates.ioIntermittent timeouts on API endpoints

Ccrates.ioIntermittent timeouts on API endpoints

TrelloTrello is slow or unavailable

Trello reported a critical outage affecting API, Trello.com, and web and mobile apps, with slow or unavailable service starting 2024-02-13 09:30 UTC and resolving 2024-02-13 11:46 UTC (computed duration: 2 h 15 min). The postmortem cited long-running database queries slowing API responses; a fix was deployed and service restored.

ngrokOAuth, OIDC, and SAML modules for Edges degraded

KeeperAustralia data center vault login errors

Keeper reported critical vault login errors affecting Keeper Admin Console (AU) and Keeper Web Vault (AU), starting 2024-02-06 02:44 UTC and resolved at 2024-02-06 02:47 UTC (computed duration: 2 min). The vendor attributed API errors to an AWS ElastiCache error in AU (Sydney); restarting the affected resource cleared the errors.

WarpWarp Blocked By CrowdStrike

PostmanElevated error rates in multiple features

Postman reported a major incident affecting login and multiple workspace features across Postman Login, Postman Platform on Browser, and Postman Platform on Desktop. It started at 2024-02-05 11:49 UTC and was resolved at 2024-02-05 12:29 UTC, lasting 39 min; Postman reported rolling out an immediate fix and monitoring stability.

PPipedreamPipedream API down, affecting https://pipedream.com

Pipedream reported a critical API outage affecting https://pipedream.com and the REST API, starting at 2024-02-03 22:42 UTC. The incident lasted 4 min and was resolved at 2024-02-03 22:47 UTC; no cause or fix was stated.

Jupyter (Anaconda)community.anaconda.cloud is not loading correctly

community.anaconda.cloud (Discourse) was not loading correctly in a major outage that started at 2024-01-31 17:13 UTC and was resolved at 2024-01-31 20:49 UTC, lasting 3 h 35 min. Anaconda investigated the failure and engaged with the vendor; no cause or specific fix was stated.

SSecureframeEmployee onboarding redirect issues

Secureframe reported a major Web Application incident preventing employee onboarding from loading for customers with policies enabled and trainings disabled. It started at 2024-01-29 16:35 UTC and lasted 3 h 12 min, resolving at 2024-01-29 19:47 UTC; a fix was implemented at 01-29 19:43 UTC.

LLightdashIssues with deploying and creating preview environments

Lightdash reported a major incident affecting EU and US Cloud, causing empty Explores during deployments or Preview environment creation. It ran from 2024-01-29 15:40 UTC to 2024-01-29 17:08 UTC (1 h 28 min). Lightdash identified release 0.956.4, reverted Cloud instances, released a fix, and recommended self-hosted users upgrade to 0.965.6 and browser users hard-refresh.

OpsgenieMajor outages in heartbeat services pinging via email and email integration in EU region

Opsgenie reported a major EU outage affecting heartbeat updates via email and incoming email integrations, rejected from 13:45 UTC due to faulty domain configurations; other heartbeat functionality and integrations remained operational. The incident ran from 2024-01-29 15:08 UTC to 2024-01-29 15:29 UTC (21 min), with service restored after changes were reverted.

MapboxAuthentication errors on Matrix, Optimization, and Directions APIs

Mapbox reported a major incident involving HTTP 401 authentication errors for some Matrix, Optimization, and Directions API users in some regions. The recorded start is 2024-01-25 23:10 UTC and resolution is 2024-01-25 12:30 UTC; these timestamps conflict, and duration is unknown. Mapbox said the issue was identified and resolved.

Jupyter (Anaconda)Anaconda.cloud login issues

Some users could not log in to anaconda.cloud during a major-impact incident starting 2024-01-25 18:18 UTC and resolved at 2024-01-25 21:44 UTC, lasting 3 h 25 min. The vendor completed a fix and confirmed all users should be able to log in.

SStitchWeb App Outage

Stitch reported a major outage of app.stitchdata.com starting at 2024-01-24 16:34 UTC and resolved at 2024-01-24 17:40 UTC, lasting 1 h 5 min. The Web App was unresponsive; Stitch reported deploying a solution and confirmed data pipelines were not impacted.

LLightstep (ServiceNow Cloud Observability)span ingestion for customers using Cloud Observability operated microsats failing

GGrainApplication instability

Grain Web App experienced major application instability with a high rate of errors after a recent code deploy, from 2024-01-19 17:18 UTC to 2024-01-19 18:43 UTC (1 h 25 min). The team issued a hot patch and marked the incident resolved after observing no further instability.

GustoWe are aware of potential error messages that customers may be experiencing when attempting to run payroll. Our teams are working to resolve this as a top priority. Thank you for your patience at this time.

Gusto reported a critical incident involving potential error messages when running payroll, affecting the API and Gusto.com website. It started at 2024-01-17 21:23 UTC and was resolved at 2024-01-17 22:21 UTC, lasting 57 min; no cause or specific fix was stated.

GustoWe are aware of potential error messages that customers may be experiencing when accessing Gusto.com. Our teams are working to resolve this as a top priority. Thank you for your patience at this time.

Customers could encounter error messages when accessing Gusto.com; Gusto marked the incident critical and listed the API and Gusto.com website as affected. It started at 2024-01-17 17:37 UTC and was resolved at 2024-01-17 17:40 UTC, lasting 3 min; no cause or fix was stated.

Travis CImacOS builds failing

Travis CI reported a major incident affecting Builds Processing - Mac Builds, with macOS builds failing from 2024-01-17 08:07 UTC to 2024-01-17 11:15 UTC, lasting 3 h 8 min. A fix was implemented at 08:24 UTC, followed by monitoring before resolution.

GustoWe are aware of potential error messages that customers may be experiencing when accessing Gusto. Our teams are working to resolve this as a top priority. Thank you for your patience at this time.

Gusto reported a critical incident affecting its API and Gusto.com website, with customers potentially seeing errors when accessing Gusto. It started at 2024-01-16 19:10 UTC and was resolved at 2024-01-16 21:32 UTC, lasting 2 h 21 min; no cause or specific fix was stated.

GustoWe are currently experiencing higher than normal call volume. During this time, please leverage help center for support.

Gusto reported higher than normal call volume affecting its Phone System, with critical impact, and directed customers to the help center for support. The incident started at 2024-01-10 23:02 UTC and was resolved at 2024-01-11 01:01 UTC, lasting 1 h 58 min; no cause or fix was stated.

PPipedreamv1 email sources not emitting events

Email sources tied to Pipedream v1 workflows failed to emit events, affecting email-triggered workflows. The vendor rated the incident major; it started at 2024-01-06 21:00 UTC and was resolved at 2024-01-06 23:31 UTC, with a reported duration of 2 h 30 min. No cause or fix was stated.

PulumiResource search timeouts

Pulumi reported a major incident in which resource search timed out, affecting API, Console, Insights, and Search. It started at 2024-01-05 20:00 UTC and was resolved at 2024-01-05 21:37 UTC, lasting 1 h 36 min. Pulumi said it identified the issue and search should be functioning normally; no specific cause or fix was provided.

GustoWe are currently experiencing higher than normal call volume. During this time, please leverage help center for support.

Gusto reported higher-than-normal call volume affecting its Phone System, rated critical, and directed customers to its help center for support. The incident started at 2024-01-03 19:42 UTC and was resolved at 2024-01-03 22:00 UTC, lasting 2 h 17 min; no cause or fix was stated.

KeeperVault login errors (Resolved)

Keeper reported critical Vault login errors, starting and resolved at 2024-01-01 15:30 UTC, with a recorded duration of 0 min. Keeper stated that AWS RDS cluster instances automatically rebooted at approximately 04:27 UTC and that it was investigating with AWS; no fix was stated.

Ccrates.ioFailing requests to crates.io

CockroachDBFailures on cluster operations

CockroachDB reported a major incident affecting AWS cluster operations, including creation and modification, starting 2023-12-22 02:01 UTC and lasting 24 h 45 min. Impact later narrowed to vertical scaling failures and delays. Resolved at 2023-12-23 02:46 UTC, the vendor stated all cluster operations should succeed and monitoring continued.

PPandaDocCan't create, view and edit documents

PandaDoc reported a critical outage preventing document creation, viewing and editing, with API, integrations and other US & Global components affected. It started at 2023-12-21 12:58 UTC and was resolved at 2023-12-21 19:00 UTC, lasting 6 h 1 min; PandaDoc reported normal application use restored during monitoring but disclosed no cause or specific fix.

PPandaDocCan't create documents

PandaDoc reported a critical outage preventing document creation, affecting Creating and editing documents and US & Global - Creating and editing documents. It started at 2023-12-20 11:50 UTC and was resolved at 2023-12-20 12:06 UTC; the computed duration was 15 min. PandaDoc reported implementing a fix and confirmed normal operation.

GGrainZoom join failures

Grain reported a critical outage starting 2023-12-19 15:33 UTC: Zoom blocked Grain Recorder from joining approximately 75% of calls. A temporary mitigation appeared successful; Grain later worked with Zoom to deploy a proper fix, reporting no further issues. The incident was resolved 2024-01-19 17:40 UTC, with a recorded duration of 31 days 2 h.