What vendors changed

PPandaDocDocuments creation via API does not work

PandaDoc reported a major API outage that prevented document creation, starting at 2023-02-13 17:23 UTC and resolved at 2023-02-13 17:39 UTC, lasting 16 min. The vendor confirmed users could create documents via API again; no cause was stated.

MezmoSearches are running slowly

Mezmo reported major Search degradation starting 2023-02-10 18:11 UTC, resolved at 2023-02-10 18:14 UTC (computed duration: 2 min). Searches were slow or returned no results; no data was lost and ingestion continued. The postmortem lists an earlier start of 16:45 UTC; normal search speeds were restored.

PPandaDocUnable to work on the documents

PandaDoc reported a critical outage preventing users from working with documents, affecting the application and document sending/opening. It started at 2023-02-08 11:46 UTC and was resolved at 2023-02-08 12:49 UTC, lasting 1 h 2 min. The vendor reported document access restored at 12:33 UTC and monitored performance before closing the incident.

JJotformJotform Outage

Jotform reported a major outage starting at 2023-02-07 15:00 UTC and marked resolved at 2023-02-07 15:00 UTC, with a computed duration of 0 min. The notice did not specify affected features, a cause, or a fix.

MezmoIntermittent delays loading Web UI and running searches

Mezmo reported a major degradation affecting Livetail, Search and Web App from 2023-02-06 20:48 UTC to 2023-02-07 02:07 UTC, lasting 5 h 18 min. Searches were slow or returned no results, and the Web UI was intermittently unresponsive; all search data access was restored. No data was lost, and ingestion was not halted.

DDailyIssues connecting to calls.

MattermostCloudflare API issues are causing Cloud Sign-Ups to fail.

Mattermost reported a major outage causing cloud workspace sign-ups to fail due to Cloudflare API issues. It started at 2023-02-01 02:45 UTC and resolved at 2023-02-01 04:46 UTC, lasting 2 h 0 min. Mattermost reported that Cloudflare had implemented a fix and monitored the results before resolution.

TTabnineNo code completions

Tabnine reported a major outage affecting Cloud Advanced Completions: users could not get code completions. It started at 2023-01-31 17:45 UTC and was resolved at 2023-01-31 18:57 UTC, lasting 1 h 11 min. Tabnine identified corrupted configuration on its inference-engine gateway and restored service after deploying a configuration change to all servers.

DagsterDagster Cloud sensors and runs failing

Dagster Cloud reported a major API incident from 2023-01-27 19:14 UTC to 2023-01-27 20:50 UTC, lasting 1 h 36 min. A GraphQL-layer error caused sensors and runs using the k8s_job_executor to fail; a fix was implemented and the incident was resolved.

Ccrates.iostatic.crates.io not resolving

TTwingateTwingate Service Incident

LLightstep (ServiceNow Cloud Observability)Alert evaluation delays

FFireboltUI issue - reporting engines states

Firebolt reported a major UI incident affecting engine-state reporting across multiple accounts and engines, starting 2023-01-19 15:27 UTC and resolved 2023-01-19 18:43 UTC; the recorded duration was 3 h 15 min. Firebolt said the issue appeared UI-only, advised customers not to restart or stop engines, and applied a fix across all engines.

LLogRocketPerformance issues with primary search database

LogRocket reported a major primary search database degradation affecting session searches, metric charts, issues and alerting; session data collection was unaffected. It started at 2023-01-18 15:41 UTC and resolved at 2023-01-20 14:00 UTC, lasting 46 h 18 min. LogRocket applied a fix and processed event backlogs, reporting no further degradation after mitigation.

DDopplerUnable to sync secrets, webhooks, and activity log notifications

Doppler reported a major partial outage blocking secret syncs, webhooks, activity-log notifications, and API-client secret updates. Records list a start of 2023-01-16 20:37 UTC and resolution at 2023-01-16 19:11 UTC, while the postmortem reports 2023-01-16 19:11 UTC to 2023-01-16 20:04 UTC; duration is unknown.

DDopplerHeroku sync integrations are failing

Doppler’s major Heroku sync outage ran from 2023-01-05 00:48 UTC to 2023-01-05 18:23 UTC (17 h 34 min), blocking new and existing syncs while previously synced secrets remained available. Resolution required all workplaces to reconnect Heroku integrations from workplace Settings, triggering pending changes to sync.

NNordLayerConnectivity issues with VPN servers in South Korea

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in South Korea, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-12-28 11:06 UTC and was resolved at 2022-12-28 12:14 UTC, lasting 1 h 8 min. The service provider implemented a fix; no further connectivity issues were detected.

NNordLayerConnectivity issues with VPN servers in Lithuania

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in Lithuania, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-12-27 18:00 UTC and was resolved at 2022-12-29 07:52 UTC, lasting 37 h 52 min. The service provider implemented a fix, and no further connectivity issues were detected.

MattermostIncident with Mattermost Boards

Mattermost Boards experienced performance and availability issues affecting Cloud Workspaces and Workspaces, rated major, from 2022-12-22 15:13 UTC to 2022-12-22 17:57 UTC (2 h 44 min). Mattermost applied a mitigation and confirmed resolution; all other services were reported working as expected.

Travis CISynchronization with GitHub fails

A major Travis CI GitHub synchronization outage prevented new repository activation and build triggering. It started 2022-12-20 14:13 UTC and was resolved 2022-12-20 15:44 UTC (1 h 30 min). Travis CI drained the pending GitHub API queue; affected customers may need to reinstall the Travis CI GitHub App.

DDremioSome projects unavailable

Some Dremio projects were unavailable in a major incident affecting Dremio Cloud - US - Site Availability and Site Availability. It started at 2022-12-15 16:35 UTC and was resolved at 2022-12-15 18:30 UTC, lasting 1 h 54 min. Dremio applied mitigations and monitored for further issues before confirming resolution.

NNordLayerConnectivity issues with VPN servers in Ireland

NordLayer reported a major incident affecting dedicated VPN server connectivity in Ireland, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-12-15 10:40 UTC and was resolved at 2022-12-15 11:30 UTC, lasting 49 min. The service provider implemented a fix, and NordLayer reported no further connectivity issues.

MMailjetSome HTTPS personalized tracking links currently not working

MattermostIncident with Community

Mattermost Community experienced degraded availability with major impact from 2022-12-09 07:53 UTC to 2022-12-09 14:03 UTC, lasting 6 h 9 min. Mattermost applied a mitigation at 12-09 08:15 UTC and later confirmed resolution; no cause was stated.

LLightstep (ServiceNow Cloud Observability)Failures ingesting metric points and displaying streams time series data

MattermostIncident with Community

Mattermost reported major degraded performance for Community from 2022-12-08 14:10 UTC to 2022-12-08 15:13 UTC, lasting 1 h 3 min. A mitigation was applied at 14:24 UTC, and the incident was subsequently confirmed resolved; no specific cause was stated.

MattermostIncident with Community

Mattermost reported a critical incident affecting Community availability from 2022-12-07 12:40 UTC to 2022-12-07 13:54 UTC, lasting 1 h 14 min. The vendor applied mitigations and confirmed resolution; no specific cause was stated.

MattermostCommunity Outage

Mattermost’s Community component experienced a major outage starting at 2022-12-06 16:00 UTC, lasting 25 min until resolution at 2022-12-06 16:25 UTC. A fix was implemented at 2022-12-06 16:16 UTC; no cause was stated.

MattermostCommunity Outage

Mattermost reported a major Community outage starting 2022-12-02 20:13 UTC, affecting some users’ ability to load or send messages. It lasted 1 h 40 min and was resolved at 2022-12-02 21:53 UTC after a fix was implemented; Boards remained disabled for the weekend pending reassessment the following week.

OpsgenieObserving delays in incoming webhook integration processing in EU region

Opsgenie reported major incoming webhook integration processing delays in the EU region starting 2022-12-02 19:09 UTC, lasting 42 min until resolution at 2022-12-02 19:51 UTC. A fix was implemented by 12-02 19:23 UTC; customers using affected integrations could experience processing delays.

MMailjetHTTPS personalized tracking links currently not working

LLightstep (ServiceNow Cloud Observability)Lightstep web application is unavailable

SStitchUser login issues

Stitch reported a major incident preventing users from logging into app.stitchdata.com, starting at 2022-12-01 09:34 UTC and resolved at 2022-12-01 15:54 UTC after 6 h 20 min. Fixes were implemented by 14:30 UTC, with possible lingering issues monitored until resolution.

SSecureframeSecureframe Not Reachable

Secureframe reported a major outage affecting its API and Web Application, preventing customers from loading part or all of Secureframe. It started at 2022-11-30 23:15 UTC and was resolved at 2022-12-01 00:05 UTC, lasting 49 min. Secureframe attributed the outage to an upstream provider, which resolved the issue.

DDailyIntermittent issues starting cloud recordings and livestreams

NNordLayerConnectivity issues with VPN servers in Brazil

NordLayer reported a major outage involving connectivity issues with dedicated VPN servers in Brazil, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-11-22 12:58 UTC and was resolved at 2022-11-22 14:56 UTC, lasting 1 h 57 min. The service provider implemented a fix, and NordLayer reported no further connectivity issues.

FFireboltEngines fail to start

Firebolt engines failed to start during a critical-impact incident beginning 2022-11-21 15:00 UTC, preventing affected customers from starting engines. The incident lasted 48 min and was resolved at 2022-11-21 15:49 UTC; Firebolt did not state a cause or specific fix.

Metabaseand MMetabase CloudIssue with Metabase Cloud

Metabase Cloud suffered a critical outage blocking ingress network access to all instances, starting 2022-11-17 18:47 UTC and resolved at 2022-11-17 19:14 UTC (26 min). DNS and load balancer configurations were accidentally overwritten while provisioning new K8S clusters in AWS; Metabase restored the correct configurations and added process controls and automated checks.

FFireboltEngines fail to start in US-EAST-1

Firebolt reported a major outage preventing engines from starting in US-EAST-1, beginning 2022-11-14 10:55 UTC and resolved at 2022-11-14 11:54 UTC (59 min). A fix restored normal engine startup by 11:41 UTC; no cause was stated.

OpsgenieAndroid users of Jira, Confluence and Opsgenie app with Compromised device check feature turned on is getting locked out of their app

Opsgenie reported a major outage locking Android users out of Jira, Confluence and Opsgenie apps when Compromised device check was enabled via admin.atlassian.com. The incident started at 2022-11-11 21:02 UTC and was resolved at 2022-11-12 00:01 UTC (2 h 58 min); the vendor confirmed normal service.

StatuspageStatuspage notifications are not being sent out

Statuspage reported a critical outage affecting Email, SMS, Twitter and Webhook notifications from 2022-11-09 22:37 UTC to 2022-11-09 23:10 UTC, lasting 33 min. A fix was implemented; all notifications were processed and none were lost.

CCloseSome users cannot log into Close

Some users could not log into or load Close’s Application UI in a major incident from 2022-11-08 13:09 UTC to 2022-11-08 16:51 UTC, lasting 3 h 42 min. Close’s postmortem attributed the loading failures to a bad cached application version; a fix was implemented and the incident resolved.

NNordLayerConnectivity issues with VPN servers in India

NordLayer reported a major outage involving connectivity issues with dedicated VPN servers in India, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-11-04 13:00 UTC and was resolved at 2022-11-04 16:59 UTC, lasting 3 h 58 min. The service provider implemented a fix; NordLayer reported no further connectivity issues.

SStitchUser login issue

Stitch reported a major login outage affecting app.stitchdata.com from 2022-11-03 17:24 UTC to 2022-11-03 20:10 UTC, lasting 2 h 46 min. The issue was resolved; only web application logins were affected, while all other functions, including extractions and loads, remained operational.

DagsterScheduler and sensor disruptions

Dagster reported a critical API incident disrupting schedulers and sensors from 2022-11-02 19:04 UTC to 2022-11-02 20:34 UTC, lasting 1 h 30 min. Dagster reported an agent request queue backend upgrade at the start; a fix was implemented by 2022-11-02 20:29 UTC, and the incident was resolved.

PPandaDocDocument can't be created from the templates

PandaDoc reported a critical outage preventing users from creating documents from templates, starting at 2022-10-28 15:26 UTC and resolved at 2022-10-28 17:55 UTC, lasting 2 h 29 min. The vendor confirmed document creation was restored but did not state a cause or describe the fix.

ShopifySome merchants are encountering errors within their admin

Shopify reported a major outage affecting admin access with 530 Authentication Errors, Plus Admin and POS App logins, and some storefront images. It started 2022-10-25 15:50 UTC and was resolved 2022-10-25 18:31 UTC, lasting 2 h 41 min. Shopify worked with partners and implemented a fix; some admins and storefronts experienced downtime.

PyPI (Python Infrastructure)PyPI Backends are unavailable.

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage causing degraded availability for a small number of workspaces, with Cloud Sign-Up, Cloud Workspaces and Sign-Up listed as affected components. It ran from 2022-10-14 14:17 UTC to 2022-10-14 16:03 UTC, lasting 1 h 46 min; a fix was implemented at 10-14 15:46 UTC.