What vendors changed

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage causing degraded availability for a small number of workspaces, with Cloud Sign-Up, Cloud Workspaces and Sign-Up listed as affected components. It ran from 2022-10-14 14:17 UTC to 2022-10-14 16:03 UTC, lasting 1 h 46 min; a fix was implemented at 10-14 15:46 UTC.

DagsterServerless agent outage

Dagster reported a major Serverless agent outage that prevented some Serverless agents from launching runs. It started and was resolved at 2022-10-13 23:00 UTC, with a computed duration of 0 min; Dagster identified and rolled back the change responsible.

Metabaseand MMetabase CloudIssues on our Cloud

Metabase reported a critical incident causing downtime on the Metabase Cloud Platform starting at 2022-10-11 17:43 UTC. Service was restored at 2022-10-11 17:53 UTC; the computed duration was 9 min. No cause or specific fix was stated.

NNordLayerConnectivity issues with London VPN servers

NordLayer reported a major outage involving connectivity issues on London dedicated and shared VPN servers, starting 2022-10-10 10:20 UTC and resolved at 2022-10-10 14:00 UTC, lasting 3 h 40 min. The service provider implemented a fix, and NordLayer confirmed all servers were back online.

LLightstep (ServiceNow Cloud Observability)Lightstep web application is unavailable

DagsterDagster Cloud Scheduler Outage

Dagster reported a major Cloud scheduler outage affecting the API component, starting at 2022-10-06 23:33 UTC and resolved at 2022-10-06 23:44 UTC (computed duration: 10 min). The scheduler returned to normal operation; no cause was stated.

CodaAttachment and image processing delays

Coda reported a critical incident delaying attachment and image processing from 2022-09-28 18:35 UTC to 2022-09-28 19:53 UTC, lasting 1 h 17 min. An outage in its virus-scanning dependency, scanii, delayed making uploads available to collaborators, although uploads were stored successfully. The incident was resolved.

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage affecting a small number of Cloud Workspaces, starting at 2022-09-21 15:08 UTC and resolved at 2022-09-21 16:38 UTC, lasting 1 h 29 min. Mattermost said it identified the root cause and mitigated the issue, but did not specify the cause or fix.

LLightstep (ServiceNow Cloud Observability)ingestion errors for certain customers

StatuspageBilling page inaccessible for some customers

Some Statuspage customers could not access the Billing page in a major-impact incident that started at 2022-09-12 16:06 UTC and was resolved at 2022-09-12 20:49 UTC, lasting 4 h 43 min. Statuspage identified the issue and implemented a fix, reporting that access should be restored for all customers at 09-12 17:03 UTC.

MMailjetTemplate builder reports into 500 error page

MMailjetIntermittent failures accessing templates

MMailjetAPI V3 Error Reports

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage affecting a small number of Cloud Workspaces, starting at 2022-08-31 13:41 UTC and resolved at 2022-08-31 18:53 UTC, with a reported duration of 5 h 11 min. The vendor identified the root cause without detailing it and reported full mitigation before resolution.

SSecureframeLogin to Secureframe failing for some customers

Some customers could not log in to Secureframe’s Web Application during a major-impact incident that started at 2022-08-31 12:16 UTC. Secureframe reported resolution at 2022-08-31 12:24 UTC; the recorded duration was 7 min. No cause or specific fix was stated.

PPandaDocClients have issues with creating templates/documents

PandaDoc reported a major incident affecting template/document creation in its Application and Web application, starting at 2022-08-30 16:00 UTC and resolved at 2022-08-30 17:33 UTC, lasting 1 h 33 min. Users were confirmed able to create templates/documents again; no cause was stated.

DDopplerSecrets unable to be read

Doppler reported a critical outage preventing secrets from being read via the API and Dashboard, starting 2022-08-29 00:38 UTC and resolved at 2022-08-29 01:38 UTC (59 min). A Redis server ran out of memory; Doppler increased its allotted memory and said it would add resource-usage alerting.

PyPI (Python Infrastructure)docs.python.org backend down

PPandaDocPandaDoc Performance Degradation

PandaDoc reported a major performance degradation causing slow loading and errors when loading and creating templates/documents for some users. It started at 2022-08-22 15:13 UTC and was resolved at 2022-08-22 17:31 UTC, lasting 2 h 18 min; PandaDoc confirmed the application was fully operational but stated no cause or specific fix.

LLightning AIApps do not start with the latest framework release

DagsterElevated error rates on Dagster Cloud

Dagster Cloud reported elevated error rates with major impact. The incident was marked resolved, but its listed resolution (2022-08-19 15:39 UTC) precedes its start (2022-08-19 16:04 UTC); duration is unknown. The only update said Dagster was investigating, with no cause or fix stated.

FFlagsmithMajor Core API outage

JJotformJotform Outage

Jotform reported a major outage blocking most users from Jotform Global, Jotform API, Jotform Europe, and HIPAA. Recorded start and resolution were 2022-08-17 17:00 UTC (computed duration: 0 min), although the update described a 30-minute outage. The vendor said database connectivity problems during planned maintenance were immediately fixed.

FFireboltStart engine may fail

Firebolt reported a major incident in which engines could fail to start, beginning 2022-08-10 11:52 UTC and resolved at 2022-08-10 12:50 UTC after 57 min. Recovery was reported for US-EAST-, then EU-WEST-1, before all regions returned to operational status.

CodaWrites through forms are not being reflected within parent doc.

Coda reported a major API degradation starting 2022-08-02 23:26 UTC: writes via the API were delayed, leaving form submissions invisible in parent docs. It was resolved at 2022-08-03 00:04 UTC after 38 min; Coda said prior submissions should now be visible and form submissions should work normally.

NNordLayerElevated levels of sign-in errors

NordLayer reported a major sign-in outage affecting Login/SSO, NordLayer applications and the NordLayer Control Panel, starting 2022-07-27 10:40 UTC and lasting 1 h 12 min. A fix was implemented by 2022-07-27 11:07 UTC; the incident was resolved at 2022-07-27 11:53 UTC, with application access fully restored.

CCourierMessage Send Delays for Legacy Segment to Send Message Integration

LLightstep (ServiceNow Cloud Observability)Ingestion and web UI outage

StatuspageErrors accessing manage portal

Statuspage reported a major incident involving errors accessing the manage portal, starting and resolving at 2022-07-21 17:20 UTC (0 min). The vendor attributed the errors to a recent deploy and restored normal traffic by rolling back to a previous code version.

StatuspageElevated Errors for New SMS Subscriptions

Statuspage reported a major incident involving elevated errors for new SMS subscriptions, affecting Hosted Pages - SMS Subscription and SMS Subscription. It started at 2022-07-21 16:37 UTC and was resolved at 2022-07-21 16:49 UTC, lasting 12 min; no cause or specific fix was stated.

LLightstep (ServiceNow Cloud Observability)Failing to ingest some span data

FFlagsmithCore API is not responding

MezmoSome customers' logs are not currently being processed

Mezmo reported a major log-ingestion degradation starting 2022-06-30 21:36 UTC, resolved at 2022-06-30 22:11 UTC (35 min). A downstream parser discarded some customers’ accepted logs, which were not retained; newly submitted logs were processing and being retained at resolution.

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage affecting a small number of Cloud Workspaces from 2022-06-24 06:52 UTC to 2022-06-24 07:52 UTC, lasting 1 h 0 min. A cluster upgrade operation degraded database connectivity; Mattermost confirmed resolution and all workspaces returning to normal.

FFireboltErrors Logging In

Some users could not log into the Firebolt UI, receiving “Failed to fetch feature flag param” errors. Firebolt classified the incident as major; it started at 2022-06-22 13:20 UTC and was resolved at 2022-06-22 14:13 UTC, lasting 53 min.

MattermostNew workspaces delay creation

Mattermost reported major delays creating new workspaces affecting Cloud Sign-Up and Sign-Up, starting 2022-06-22 04:29 UTC and resolved at 2022-06-22 05:48 UTC (1 h 18 min). The vendor identified a communication issue with its DNS provider and implemented a fix.

TTwingateDocumentation site is unavailable in some regions

ShopifyWe are investigating an issue preventing web traffic to our servers

Shopify reported a major outage preventing web traffic to its servers, affecting Admin, API & Mobile, Checkout, Point of Sale, Reports and Dashboards, Storefront, Support, and Third party services. It ran from 2022-06-21 06:59 UTC to 2022-06-21 08:11 UTC, lasting 1 h 11 min. Shopify applied a fix and confirmed normal traffic with no further errors.

DDopplerServices responding with 500 Internal Server ErrorEvent

Doppler’s API (api.doppler.com) and Dashboard (dashboard.doppler.com) returned 500 Internal Server Error responses during a critical outage from 2022-06-21 06:33 UTC to 2022-06-21 07:22 UTC, lasting 49 min. Doppler attributed the issue to a widespread Cloudflare outage and reported implementing a fix before resolution.

MMailjetNot possible to set time for Scheduling Campaign

NNordLayerLos Angeles connectivity issues (shared and dedicated servers)

NordLayer reported a critical outage affecting Los Angeles shared and dedicated servers, including Shared Gateways and Private Gateways (Servers with dedicated IPs). It started at 2022-06-10 20:13 UTC and lasted 38 min, with resolution reported at 2022-06-10 20:52 UTC after the service provider implemented a fix; all Los Angeles servers were back online.

SStitchDegraded service in AWS us-east-1 affecting both extractions and loads

Stitch reported major service degradation in AWS us-east-1 affecting extractions and loads, with api.stitchdata.com, app.stitchdata.com and AWS kms-us-east-1 listed as affected components. It started at 2022-06-10 04:01 UTC and lasted 9 h 23 min, resolving at 2022-06-10 13:25 UTC. Stitch reported implementing a fix at 04:26 UTC but did not state a cause.

MMailjetLogin Issues with Mailjet Application

FFireboltstarting & stopping engines takes long time.

Firebolt reported a major degradation affecting engine starts and stops from 2022-06-08 08:18 UTC to 2022-06-08 11:08 UTC, lasting 2 h 50 min. Starting or stopping engines could be slow or fail; a production fix restored those operations at 09:37 UTC, and Firebolt confirmed normal service at resolution.

FFireboltstarting & stopping engines takes long time.

Firebolt reported a major degradation affecting Engines: starting and stopping engines took a long time. The incident ran from 2022-06-08 07:11 UTC to 2022-06-08 07:35 UTC, lasting 24 min. Firebolt said it identified the root cause without specifying it, then confirmed the issue was fixed and engines could be started and stopped.

MezmoLog lines not ingested

Mezmo reported a critical log-ingestion outage across all sources from 2022-06-02 21:44 UTC to 2022-06-02 22:05 UTC (21 min). Newly submitted logs were never ingested, making them unavailable for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving. Ingestion resumed and all services were fully operational.

FFireboltLogin issue

Firebolt reported a major login outage affecting its API and Web Application, starting 2022-05-25 16:38 UTC and resolved at 2022-05-25 17:50 UTC (1 h 11 min). Customers experienced transient login issues; Firebolt said it identified the immediate cause and was implementing a long-term fix before confirming services were back to normal.

Travis CISome build requests are not being triggered.

Travis CI reported a major incident in which some build requests triggered through the API and Git push were not processed. The recorded start was 2022-05-17 06:21 UTC; it was marked resolved at 2022-05-17 03:00 UTC, but duration is unknown. The update said the root cause was still under investigation; no fix was stated.

TTwingateIssue with Twingate