What vendors changed

MMailjetIntermittent failures accessing templates

MMailjetAPI V3 Error Reports

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage affecting a small number of Cloud Workspaces, starting at 2022-08-31 13:41 UTC and resolved at 2022-08-31 18:53 UTC, with a reported duration of 5 h 11 min. The vendor identified the root cause without detailing it and reported full mitigation before resolution.

SSecureframeLogin to Secureframe failing for some customers

Some customers could not log in to Secureframe’s Web Application during a major-impact incident that started at 2022-08-31 12:16 UTC. Secureframe reported resolution at 2022-08-31 12:24 UTC; the recorded duration was 7 min. No cause or specific fix was stated.

PPandaDocClients have issues with creating templates/documents

PandaDoc reported a major incident affecting template/document creation in its Application and Web application, starting at 2022-08-30 16:00 UTC and resolved at 2022-08-30 17:33 UTC, lasting 1 h 33 min. Users were confirmed able to create templates/documents again; no cause was stated.

DDopplerSecrets unable to be read

Doppler reported a critical outage preventing secrets from being read via the API and Dashboard, starting 2022-08-29 00:38 UTC and resolved at 2022-08-29 01:38 UTC (59 min). A Redis server ran out of memory; Doppler increased its allotted memory and said it would add resource-usage alerting.

PyPI (Python Infrastructure)docs.python.org backend down

PPandaDocPandaDoc Performance Degradation

PandaDoc reported a major performance degradation causing slow loading and errors when loading and creating templates/documents for some users. It started at 2022-08-22 15:13 UTC and was resolved at 2022-08-22 17:31 UTC, lasting 2 h 18 min; PandaDoc confirmed the application was fully operational but stated no cause or specific fix.

LLightning AIApps do not start with the latest framework release

DagsterElevated error rates on Dagster Cloud

Dagster Cloud reported elevated error rates with major impact. The incident was marked resolved, but its listed resolution (2022-08-19 15:39 UTC) precedes its start (2022-08-19 16:04 UTC); duration is unknown. The only update said Dagster was investigating, with no cause or fix stated.

FFlagsmithMajor Core API outage

JJotformJotform Outage

Jotform reported a major outage blocking most users from Jotform Global, Jotform API, Jotform Europe, and HIPAA. Recorded start and resolution were 2022-08-17 17:00 UTC (computed duration: 0 min), although the update described a 30-minute outage. The vendor said database connectivity problems during planned maintenance were immediately fixed.

FFireboltStart engine may fail

Firebolt reported a major incident in which engines could fail to start, beginning 2022-08-10 11:52 UTC and resolved at 2022-08-10 12:50 UTC after 57 min. Recovery was reported for US-EAST-, then EU-WEST-1, before all regions returned to operational status.

CodaWrites through forms are not being reflected within parent doc.

Coda reported a major API degradation starting 2022-08-02 23:26 UTC: writes via the API were delayed, leaving form submissions invisible in parent docs. It was resolved at 2022-08-03 00:04 UTC after 38 min; Coda said prior submissions should now be visible and form submissions should work normally.

NNordLayerElevated levels of sign-in errors

NordLayer reported a major sign-in outage affecting Login/SSO, NordLayer applications and the NordLayer Control Panel, starting 2022-07-27 10:40 UTC and lasting 1 h 12 min. A fix was implemented by 2022-07-27 11:07 UTC; the incident was resolved at 2022-07-27 11:53 UTC, with application access fully restored.

CCourierMessage Send Delays for Legacy Segment to Send Message Integration

LLightstep (ServiceNow Cloud Observability)Ingestion and web UI outage

StatuspageErrors accessing manage portal

Statuspage reported a major incident involving errors accessing the manage portal, starting and resolving at 2022-07-21 17:20 UTC (0 min). The vendor attributed the errors to a recent deploy and restored normal traffic by rolling back to a previous code version.

StatuspageElevated Errors for New SMS Subscriptions

Statuspage reported a major incident involving elevated errors for new SMS subscriptions, affecting Hosted Pages - SMS Subscription and SMS Subscription. It started at 2022-07-21 16:37 UTC and was resolved at 2022-07-21 16:49 UTC, lasting 12 min; no cause or specific fix was stated.

LLightstep (ServiceNow Cloud Observability)Failing to ingest some span data

FFlagsmithCore API is not responding

MezmoSome customers' logs are not currently being processed

Mezmo reported a major log-ingestion degradation starting 2022-06-30 21:36 UTC, resolved at 2022-06-30 22:11 UTC (35 min). A downstream parser discarded some customers’ accepted logs, which were not retained; newly submitted logs were processing and being retained at resolution.

MattermostPartial outage affecting a small number of workspaces

Mattermost reported a major partial outage affecting a small number of Cloud Workspaces from 2022-06-24 06:52 UTC to 2022-06-24 07:52 UTC, lasting 1 h 0 min. A cluster upgrade operation degraded database connectivity; Mattermost confirmed resolution and all workspaces returning to normal.

FFireboltErrors Logging In

Some users could not log into the Firebolt UI, receiving “Failed to fetch feature flag param” errors. Firebolt classified the incident as major; it started at 2022-06-22 13:20 UTC and was resolved at 2022-06-22 14:13 UTC, lasting 53 min.

MattermostNew workspaces delay creation

Mattermost reported major delays creating new workspaces affecting Cloud Sign-Up and Sign-Up, starting 2022-06-22 04:29 UTC and resolved at 2022-06-22 05:48 UTC (1 h 18 min). The vendor identified a communication issue with its DNS provider and implemented a fix.

TTwingateDocumentation site is unavailable in some regions

ShopifyWe are investigating an issue preventing web traffic to our servers

Shopify reported a major outage preventing web traffic to its servers, affecting Admin, API & Mobile, Checkout, Point of Sale, Reports and Dashboards, Storefront, Support, and Third party services. It ran from 2022-06-21 06:59 UTC to 2022-06-21 08:11 UTC, lasting 1 h 11 min. Shopify applied a fix and confirmed normal traffic with no further errors.

DDopplerServices responding with 500 Internal Server ErrorEvent

Doppler’s API (api.doppler.com) and Dashboard (dashboard.doppler.com) returned 500 Internal Server Error responses during a critical outage from 2022-06-21 06:33 UTC to 2022-06-21 07:22 UTC, lasting 49 min. Doppler attributed the issue to a widespread Cloudflare outage and reported implementing a fix before resolution.

MMailjetNot possible to set time for Scheduling Campaign

NNordLayerLos Angeles connectivity issues (shared and dedicated servers)

NordLayer reported a critical outage affecting Los Angeles shared and dedicated servers, including Shared Gateways and Private Gateways (Servers with dedicated IPs). It started at 2022-06-10 20:13 UTC and lasted 38 min, with resolution reported at 2022-06-10 20:52 UTC after the service provider implemented a fix; all Los Angeles servers were back online.

SStitchDegraded service in AWS us-east-1 affecting both extractions and loads

Stitch reported major service degradation in AWS us-east-1 affecting extractions and loads, with api.stitchdata.com, app.stitchdata.com and AWS kms-us-east-1 listed as affected components. It started at 2022-06-10 04:01 UTC and lasted 9 h 23 min, resolving at 2022-06-10 13:25 UTC. Stitch reported implementing a fix at 04:26 UTC but did not state a cause.

MMailjetLogin Issues with Mailjet Application

FFireboltstarting & stopping engines takes long time.

Firebolt reported a major degradation affecting engine starts and stops from 2022-06-08 08:18 UTC to 2022-06-08 11:08 UTC, lasting 2 h 50 min. Starting or stopping engines could be slow or fail; a production fix restored those operations at 09:37 UTC, and Firebolt confirmed normal service at resolution.

FFireboltstarting & stopping engines takes long time.

Firebolt reported a major degradation affecting Engines: starting and stopping engines took a long time. The incident ran from 2022-06-08 07:11 UTC to 2022-06-08 07:35 UTC, lasting 24 min. Firebolt said it identified the root cause without specifying it, then confirmed the issue was fixed and engines could be started and stopped.

MezmoLog lines not ingested

Mezmo reported a critical log-ingestion outage across all sources from 2022-06-02 21:44 UTC to 2022-06-02 22:05 UTC (21 min). Newly submitted logs were never ingested, making them unavailable for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving. Ingestion resumed and all services were fully operational.

FFireboltLogin issue

Firebolt reported a major login outage affecting its API and Web Application, starting 2022-05-25 16:38 UTC and resolved at 2022-05-25 17:50 UTC (1 h 11 min). Customers experienced transient login issues; Firebolt said it identified the immediate cause and was implementing a long-term fix before confirming services were back to normal.

Travis CISome build requests are not being triggered.

Travis CI reported a major incident in which some build requests triggered through the API and Git push were not processed. The recorded start was 2022-05-17 06:21 UTC; it was marked resolved at 2022-05-17 03:00 UTC, but duration is unknown. The update said the root cause was still under investigation; no fix was stated.

TTwingateIssue with Twingate

DDremioQuery engines not scaling up

Dremio Cloud - US had a major incident: query engines failed to scale up on demand, causing query failures; already-running engines remained unaffected. It started at 2022-05-10 05:32 UTC and was resolved at 2022-05-10 06:49 UTC, lasting 1 h 16 min. Dremio reported a successful mitigation and confirmed engine scaling was working normally.

NNordLayerLos Angeles connectivity issues (shared and dedicated servers)

NordLayer reported a critical outage affecting Los Angeles shared and dedicated servers, including private and shared gateways, from 2022-05-04 14:13 UTC to 2022-05-04 14:51 UTC (37 min). The vendor attributed connectivity issues to a power supply issue at its service provider’s LA site and confirmed all Los Angeles servers were back online.

SStitchSome extraction jobs not starting

Some Stitch extraction jobs failed to start at their scheduled time, affecting api.stitchdata.com and app.stitchdata.com. The vendor rated the incident major; it ran from 2022-04-28 14:41 UTC to 2022-04-28 22:12 UTC, lasting 7 h 31 min. A fix was implemented at 18:04 UTC, and the incident was subsequently resolved.

NNordLayerElevated levels of sign-in errors

NordLayer reported a major incident involving elevated sign-in errors affecting Login/SSO and NordLayer applications, starting 2022-04-27 08:30 UTC and lasting 34 min. A fix was implemented at 08:42 UTC, and application access was fully restored by 2022-04-27 09:04 UTC.

FFireboltEngines failing to start in us-east-1

Firebolt reported a major outage affecting engine provisioning and startups in us-east-1 from 2022-04-19 15:18 UTC to 2022-04-19 18:04 UTC, lasting 2 h 45 min. Customers were told not to stop or restart existing engines. Firebolt attributed the disruption to an AWS issue; after its resolution, startups and restarts were permitted again, with stuck engines expected to start shortly.

MezmoAlerts not triggering

Mezmo’s critical Alerting outage ran from 2022-04-05 18:23 UTC to 2022-04-05 18:32 UTC (9 min); alerts failed for all accounts before service resumed. The postmortem mentions a parser service restart and says most alerts with trigger dates more than 15 minutes in the past were discarded.

Codacoda.io outage

Coda reported a critical site outage affecting Doc List and Docs from 2022-03-15 19:19 UTC to 2022-03-15 19:27 UTC, lasting 8 min. The incident was resolved; Coda stated it would improve its processes to prevent recurrence but provided no cause or specific fix.

LLogRocketIngestion outage

LogRocket reported a major ingestion outage affecting Alerts and Session Collection from 2022-03-04 18:42 UTC to 2022-03-04 20:44 UTC, lasting 2 h 2 min. New-data ingestion was delayed during a primary database issue; ingestion recovered at 19:26 UTC, but alerting remained impacted at 19:45 UTC before systems returned to normal.

Travis CISome builds are not being triggerred

Travis CI reported a major outage affecting Linux, Windows and Mac build processing: some builds were not triggered or returned an undetailed error when triggered. It lasted 55 min, from 2022-03-01 12:08 UTC to 2022-03-01 13:03 UTC; a fix was implemented and monitored before resolution.

DDremioQueries failing in North America AWS regions

Dremio reported a critical incident in North America AWS regions: queries failed to execute while the site remained available. It started at 2022-02-28 18:44 UTC and was resolved at 2022-02-28 19:21 UTC, lasting 36 min. Dremio applied a mitigation and then confirmed a fix was implemented.

MezmoIngestion of new logs — for Syslog only - Partial Outage

Mezmo reported major, Syslog-only log ingestion failures from 2022-02-24 22:45 UTC to resolution at 2022-02-24 23:43 UTC (57 min computed). New logs intermittently failed to ingest. The postmortem gives an earlier start—Thursday, February 18, 2022, at 00:10 UTC—and mentions the newly introduced Syslog Forwarder service.

Ccrates.iocrates.io unavailable