What vendors changed

LLogRocketPerformance issues with primary search database

LogRocket reported a major primary search database degradation affecting session searches, metric charts, issues and alerting; session data collection was unaffected. It started at 2023-01-18 15:41 UTC and resolved at 2023-01-20 14:00 UTC, lasting 46 h 18 min. LogRocket applied a fix and processed event backlogs, reporting no further degradation after mitigation.

DDopplerUnable to sync secrets, webhooks, and activity log notifications

Doppler reported a major partial outage blocking secret syncs, webhooks, activity-log notifications, and API-client secret updates. Records list a start of 2023-01-16 20:37 UTC and resolution at 2023-01-16 19:11 UTC, while the postmortem reports 2023-01-16 19:11 UTC to 2023-01-16 20:04 UTC; duration is unknown.

DDopplerHeroku sync integrations are failing

Doppler’s major Heroku sync outage ran from 2023-01-05 00:48 UTC to 2023-01-05 18:23 UTC (17 h 34 min), blocking new and existing syncs while previously synced secrets remained available. Resolution required all workplaces to reconnect Heroku integrations from workplace Settings, triggering pending changes to sync.

NNordLayerConnectivity issues with VPN servers in South Korea

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in South Korea, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-12-28 11:06 UTC and was resolved at 2022-12-28 12:14 UTC, lasting 1 h 8 min. The service provider implemented a fix; no further connectivity issues were detected.

NNordLayerConnectivity issues with VPN servers in Lithuania

NordLayer reported a major incident affecting connectivity to dedicated VPN servers in Lithuania, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-12-27 18:00 UTC and was resolved at 2022-12-29 07:52 UTC, lasting 37 h 52 min. The service provider implemented a fix, and no further connectivity issues were detected.

MattermostIncident with Mattermost Boards

Mattermost Boards experienced performance and availability issues affecting Cloud Workspaces and Workspaces, rated major, from 2022-12-22 15:13 UTC to 2022-12-22 17:57 UTC (2 h 44 min). Mattermost applied a mitigation and confirmed resolution; all other services were reported working as expected.

Travis CISynchronization with GitHub fails

A major Travis CI GitHub synchronization outage prevented new repository activation and build triggering. It started 2022-12-20 14:13 UTC and was resolved 2022-12-20 15:44 UTC (1 h 30 min). Travis CI drained the pending GitHub API queue; affected customers may need to reinstall the Travis CI GitHub App.

DDremioSome projects unavailable

Some Dremio projects were unavailable in a major incident affecting Dremio Cloud - US - Site Availability and Site Availability. It started at 2022-12-15 16:35 UTC and was resolved at 2022-12-15 18:30 UTC, lasting 1 h 54 min. Dremio applied mitigations and monitored for further issues before confirming resolution.

NNordLayerConnectivity issues with VPN servers in Ireland

NordLayer reported a major incident affecting dedicated VPN server connectivity in Ireland, including Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-12-15 10:40 UTC and was resolved at 2022-12-15 11:30 UTC, lasting 49 min. The service provider implemented a fix, and NordLayer reported no further connectivity issues.

MMailjetSome HTTPS personalized tracking links currently not working

MattermostIncident with Community

Mattermost Community experienced degraded availability with major impact from 2022-12-09 07:53 UTC to 2022-12-09 14:03 UTC, lasting 6 h 9 min. Mattermost applied a mitigation at 12-09 08:15 UTC and later confirmed resolution; no cause was stated.

LLightstep (ServiceNow Cloud Observability)Failures ingesting metric points and displaying streams time series data

MattermostIncident with Community

Mattermost reported major degraded performance for Community from 2022-12-08 14:10 UTC to 2022-12-08 15:13 UTC, lasting 1 h 3 min. A mitigation was applied at 14:24 UTC, and the incident was subsequently confirmed resolved; no specific cause was stated.

MattermostIncident with Community

Mattermost reported a critical incident affecting Community availability from 2022-12-07 12:40 UTC to 2022-12-07 13:54 UTC, lasting 1 h 14 min. The vendor applied mitigations and confirmed resolution; no specific cause was stated.

MattermostCommunity Outage

Mattermost’s Community component experienced a major outage starting at 2022-12-06 16:00 UTC, lasting 25 min until resolution at 2022-12-06 16:25 UTC. A fix was implemented at 2022-12-06 16:16 UTC; no cause was stated.

MattermostCommunity Outage

Mattermost reported a major Community outage starting 2022-12-02 20:13 UTC, affecting some users’ ability to load or send messages. It lasted 1 h 40 min and was resolved at 2022-12-02 21:53 UTC after a fix was implemented; Boards remained disabled for the weekend pending reassessment the following week.

OpsgenieObserving delays in incoming webhook integration processing in EU region

Opsgenie reported major incoming webhook integration processing delays in the EU region starting 2022-12-02 19:09 UTC, lasting 42 min until resolution at 2022-12-02 19:51 UTC. A fix was implemented by 12-02 19:23 UTC; customers using affected integrations could experience processing delays.

MMailjetHTTPS personalized tracking links currently not working

LLightstep (ServiceNow Cloud Observability)Lightstep web application is unavailable

SStitchUser login issues

Stitch reported a major incident preventing users from logging into app.stitchdata.com, starting at 2022-12-01 09:34 UTC and resolved at 2022-12-01 15:54 UTC after 6 h 20 min. Fixes were implemented by 14:30 UTC, with possible lingering issues monitored until resolution.

SSecureframeSecureframe Not Reachable

Secureframe reported a major outage affecting its API and Web Application, preventing customers from loading part or all of Secureframe. It started at 2022-11-30 23:15 UTC and was resolved at 2022-12-01 00:05 UTC, lasting 49 min. Secureframe attributed the outage to an upstream provider, which resolved the issue.

DDailyIntermittent issues starting cloud recordings and livestreams

NNordLayerConnectivity issues with VPN servers in Brazil

NordLayer reported a major outage involving connectivity issues with dedicated VPN servers in Brazil, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-11-22 12:58 UTC and was resolved at 2022-11-22 14:56 UTC, lasting 1 h 57 min. The service provider implemented a fix, and NordLayer reported no further connectivity issues.

FFireboltEngines fail to start

Firebolt engines failed to start during a critical-impact incident beginning 2022-11-21 15:00 UTC, preventing affected customers from starting engines. The incident lasted 48 min and was resolved at 2022-11-21 15:49 UTC; Firebolt did not state a cause or specific fix.

Metabaseand MMetabase CloudIssue with Metabase Cloud

Metabase Cloud suffered a critical outage blocking ingress network access to all instances, starting 2022-11-17 18:47 UTC and resolved at 2022-11-17 19:14 UTC (26 min). DNS and load balancer configurations were accidentally overwritten while provisioning new K8S clusters in AWS; Metabase restored the correct configurations and added process controls and automated checks.

FFireboltEngines fail to start in US-EAST-1

Firebolt reported a major outage preventing engines from starting in US-EAST-1, beginning 2022-11-14 10:55 UTC and resolved at 2022-11-14 11:54 UTC (59 min). A fix restored normal engine startup by 11:41 UTC; no cause was stated.

OpsgenieAndroid users of Jira, Confluence and Opsgenie app with Compromised device check feature turned on is getting locked out of their app

Opsgenie reported a major outage locking Android users out of Jira, Confluence and Opsgenie apps when Compromised device check was enabled via admin.atlassian.com. The incident started at 2022-11-11 21:02 UTC and was resolved at 2022-11-12 00:01 UTC (2 h 58 min); the vendor confirmed normal service.

StatuspageStatuspage notifications are not being sent out

Statuspage reported a critical outage affecting Email, SMS, Twitter and Webhook notifications from 2022-11-09 22:37 UTC to 2022-11-09 23:10 UTC, lasting 33 min. A fix was implemented; all notifications were processed and none were lost.

CCloseSome users cannot log into Close

Some users could not log into or load Close’s Application UI in a major incident from 2022-11-08 13:09 UTC to 2022-11-08 16:51 UTC, lasting 3 h 42 min. Close’s postmortem attributed the loading failures to a bad cached application version; a fix was implemented and the incident resolved.

NNordLayerConnectivity issues with VPN servers in India

NordLayer reported a major outage involving connectivity issues with dedicated VPN servers in India, affecting Dedicated Servers and Private Gateways (Servers with dedicated IPs). It started at 2022-11-04 13:00 UTC and was resolved at 2022-11-04 16:59 UTC, lasting 3 h 58 min. The service provider implemented a fix; NordLayer reported no further connectivity issues.

SStitchUser login issue

Stitch reported a major login outage affecting app.stitchdata.com from 2022-11-03 17:24 UTC to 2022-11-03 20:10 UTC, lasting 2 h 46 min. The issue was resolved; only web application logins were affected, while all other functions, including extractions and loads, remained operational.

DagsterScheduler and sensor disruptions

Dagster reported a critical API incident disrupting schedulers and sensors from 2022-11-02 19:04 UTC to 2022-11-02 20:34 UTC, lasting 1 h 30 min. Dagster reported an agent request queue backend upgrade at the start; a fix was implemented by 2022-11-02 20:29 UTC, and the incident was resolved.

PPandaDocDocument can't be created from the templates

PandaDoc reported a critical outage preventing users from creating documents from templates, starting at 2022-10-28 15:26 UTC and resolved at 2022-10-28 17:55 UTC, lasting 2 h 29 min. The vendor confirmed document creation was restored but did not state a cause or describe the fix.

ShopifySome merchants are encountering errors within their admin

Shopify reported a major outage affecting admin access with 530 Authentication Errors, Plus Admin and POS App logins, and some storefront images. It started 2022-10-25 15:50 UTC and was resolved 2022-10-25 18:31 UTC, lasting 2 h 41 min. Shopify worked with partners and implemented a fix; some admins and storefronts experienced downtime.

PyPI (Python Infrastructure)PyPI Backends are unavailable.

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage causing degraded availability for a small number of workspaces, with Cloud Sign-Up, Cloud Workspaces and Sign-Up listed as affected components. It ran from 2022-10-14 14:17 UTC to 2022-10-14 16:03 UTC, lasting 1 h 46 min; a fix was implemented at 10-14 15:46 UTC.

DagsterServerless agent outage

Dagster reported a major Serverless agent outage that prevented some Serverless agents from launching runs. It started and was resolved at 2022-10-13 23:00 UTC, with a computed duration of 0 min; Dagster identified and rolled back the change responsible.

Metabaseand MMetabase CloudIssues on our Cloud

Metabase reported a critical incident causing downtime on the Metabase Cloud Platform starting at 2022-10-11 17:43 UTC. Service was restored at 2022-10-11 17:53 UTC; the computed duration was 9 min. No cause or specific fix was stated.

NNordLayerConnectivity issues with London VPN servers

NordLayer reported a major outage involving connectivity issues on London dedicated and shared VPN servers, starting 2022-10-10 10:20 UTC and resolved at 2022-10-10 14:00 UTC, lasting 3 h 40 min. The service provider implemented a fix, and NordLayer confirmed all servers were back online.

LLightstep (ServiceNow Cloud Observability)Lightstep web application is unavailable

DagsterDagster Cloud Scheduler Outage

Dagster reported a major Cloud scheduler outage affecting the API component, starting at 2022-10-06 23:33 UTC and resolved at 2022-10-06 23:44 UTC (computed duration: 10 min). The scheduler returned to normal operation; no cause was stated.

CodaAttachment and image processing delays

Coda reported a critical incident delaying attachment and image processing from 2022-09-28 18:35 UTC to 2022-09-28 19:53 UTC, lasting 1 h 17 min. An outage in its virus-scanning dependency, scanii, delayed making uploads available to collaborators, although uploads were stored successfully. The incident was resolved.

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage affecting a small number of Cloud Workspaces, starting at 2022-09-21 15:08 UTC and resolved at 2022-09-21 16:38 UTC, lasting 1 h 29 min. Mattermost said it identified the root cause and mitigated the issue, but did not specify the cause or fix.

LLightstep (ServiceNow Cloud Observability)ingestion errors for certain customers

StatuspageBilling page inaccessible for some customers

Some Statuspage customers could not access the Billing page in a major-impact incident that started at 2022-09-12 16:06 UTC and was resolved at 2022-09-12 20:49 UTC, lasting 4 h 43 min. Statuspage identified the issue and implemented a fix, reporting that access should be restored for all customers at 09-12 17:03 UTC.

MMailjetTemplate builder reports into 500 error page

MMailjetIntermittent failures accessing templates

MMailjetAPI V3 Error Reports