What vendors changed

TTwingateIssue with Twingate

DDremioQuery engines not scaling up

Dremio Cloud - US had a major incident: query engines failed to scale up on demand, causing query failures; already-running engines remained unaffected. It started at 2022-05-10 05:32 UTC and was resolved at 2022-05-10 06:49 UTC, lasting 1 h 16 min. Dremio reported a successful mitigation and confirmed engine scaling was working normally.

NNordLayerLos Angeles connectivity issues (shared and dedicated servers)

NordLayer reported a critical outage affecting Los Angeles shared and dedicated servers, including private and shared gateways, from 2022-05-04 14:13 UTC to 2022-05-04 14:51 UTC (37 min). The vendor attributed connectivity issues to a power supply issue at its service provider’s LA site and confirmed all Los Angeles servers were back online.

SStitchSome extraction jobs not starting

Some Stitch extraction jobs failed to start at their scheduled time, affecting api.stitchdata.com and app.stitchdata.com. The vendor rated the incident major; it ran from 2022-04-28 14:41 UTC to 2022-04-28 22:12 UTC, lasting 7 h 31 min. A fix was implemented at 18:04 UTC, and the incident was subsequently resolved.

NNordLayerElevated levels of sign-in errors

NordLayer reported a major incident involving elevated sign-in errors affecting Login/SSO and NordLayer applications, starting 2022-04-27 08:30 UTC and lasting 34 min. A fix was implemented at 08:42 UTC, and application access was fully restored by 2022-04-27 09:04 UTC.

FFireboltEngines failing to start in us-east-1

Firebolt reported a major outage affecting engine provisioning and startups in us-east-1 from 2022-04-19 15:18 UTC to 2022-04-19 18:04 UTC, lasting 2 h 45 min. Customers were told not to stop or restart existing engines. Firebolt attributed the disruption to an AWS issue; after its resolution, startups and restarts were permitted again, with stuck engines expected to start shortly.

MezmoAlerts not triggering

Mezmo’s critical Alerting outage ran from 2022-04-05 18:23 UTC to 2022-04-05 18:32 UTC (9 min); alerts failed for all accounts before service resumed. The postmortem mentions a parser service restart and says most alerts with trigger dates more than 15 minutes in the past were discarded.

Codacoda.io outage

Coda reported a critical site outage affecting Doc List and Docs from 2022-03-15 19:19 UTC to 2022-03-15 19:27 UTC, lasting 8 min. The incident was resolved; Coda stated it would improve its processes to prevent recurrence but provided no cause or specific fix.

LLogRocketIngestion outage

LogRocket reported a major ingestion outage affecting Alerts and Session Collection from 2022-03-04 18:42 UTC to 2022-03-04 20:44 UTC, lasting 2 h 2 min. New-data ingestion was delayed during a primary database issue; ingestion recovered at 19:26 UTC, but alerting remained impacted at 19:45 UTC before systems returned to normal.

Travis CISome builds are not being triggerred

Travis CI reported a major outage affecting Linux, Windows and Mac build processing: some builds were not triggered or returned an undetailed error when triggered. It lasted 55 min, from 2022-03-01 12:08 UTC to 2022-03-01 13:03 UTC; a fix was implemented and monitored before resolution.

DDremioQueries failing in North America AWS regions

Dremio reported a critical incident in North America AWS regions: queries failed to execute while the site remained available. It started at 2022-02-28 18:44 UTC and was resolved at 2022-02-28 19:21 UTC, lasting 36 min. Dremio applied a mitigation and then confirmed a fix was implemented.

MezmoIngestion of new logs — for Syslog only - Partial Outage

Mezmo reported major, Syslog-only log ingestion failures from 2022-02-24 22:45 UTC to resolution at 2022-02-24 23:43 UTC (57 min computed). New logs intermittently failed to ingest. The postmortem gives an earlier start—Thursday, February 18, 2022, at 00:10 UTC—and mentions the newly introduced Syslog Forwarder service.

Ccrates.iocrates.io unavailable

MezmoIngestion of new logs — for Syslog only - is intermittently failing

Mezmo reported a major degradation: new-log ingestion through its Syslog endpoint intermittently failed from 2022-02-18 01:41 UTC to 2022-02-18 02:15 UTC (33 min). The incident was resolved; the postmortem attributed the failures to a code change in Syslog Forwarder, which handles Syslog ingestion.

MezmoSome logs submitted over syslog using custom ports are not being correctly formatted and are not available for Alerting, Searching, Timelines, Graphs, and Live Tail.

Mezmo reported a major Log Ingestion (Syslog) outage affecting some custom-port logs, making them unavailable for Alerting, Searching, Timelines, Graphs, and Live Tail; logs not using custom ports worked normally. It started at 2022-02-16 19:58 UTC and resolved at 2022-02-16 21:56 UTC (1 h 57 min). A fix was implemented.

StatuspagePublic API experiencing increased errors

Statuspage reported a major Public API incident in which increased load caused API errors. The incident started and was resolved at 2022-02-07 18:00 UTC, with a computed duration of 0 min.

MMailjetPartial outbound message delivery delays

LLightstep (ServiceNow Cloud Observability)Errors Ingesting Span Data

LLightstep (ServiceNow Cloud Observability)Delays receiving data from satellites

CodaPacks issues

Coda reported a critical outage affecting Packs starting 2022-01-31 23:15 UTC, resolved at 2022-02-01 00:09 UTC after 54 min. No cause or specific fix was stated.

LLightstep (ServiceNow Cloud Observability)Slack / commands not working

MezmoIngestion, Searching, Live Tail, Alerting, Graphing, and Timelines Delays

Mezmo reported a critical outage: ingestion halted, delaying Searching, Live Tail, Alerting, Graphing, and Timelines; some alerts never triggered. The incident ran from 2022-01-26 16:10 UTC to 2022-01-26 17:15 UTC, lasting 1 h 5 min. A fix restored ingestion and all services by 16:58 UTC.

FFireboltEngines may fail to start and stop

Firebolt reported a major incident affecting engine starts and stops from 2022-01-26 13:02 UTC to 2022-01-26 16:48 UTC (3 h 45 min). Ingestion on impacted engines may also have been affected. Firebolt reported a master-cluster outage that disrupted internal services; a fix was deployed and tested before resolution.

MattermostCollapse threads are not functional in part of Cloud workspaces

Collapsed threads were not functional in some Mattermost Cloud Workspaces during a major-impact incident that started at 2022-01-26 09:05 UTC and was resolved at 2022-01-26 16:04 UTC, lasting 6 h 59 min. Mattermost reported identifying the root cause and working to mitigate affected workspaces, but did not specify the cause or fix.

FFireboltLoading databases could fail

Firebolt reported a critical outage in which database loading could fail, preventing UI connections and potentially affecting API requests. It started at 2022-01-24 09:24 UTC and was resolved at 2022-01-24 13:25 UTC, lasting 4 h 0 min. A fix was implemented and monitored before resolution.

LLightstep (ServiceNow Cloud Observability)Lightstep Webapp Down

TTwingateIssue connecting to Twingate

MMailjetTemplates unavailable for some users

LLightstep (ServiceNow Cloud Observability)Lightstep Webapp and API Experiencing Issues

DDopplerUnable to connect to the Doppler dashboard and API

Doppler reported a critical outage preventing connections to its API (api.doppler.com) and Dashboard (dashboard.doppler.com), starting at 2021-12-22 16:50 UTC. A fix was implemented by 2021-12-22 16:54 UTC, and the incident was resolved at 2021-12-22 17:08 UTC; the computed duration was 17 min.

FFireboltAmazon Elastic Compute outageEvent

Firebolt reported a major Amazon Elastic Compute outage affecting AWS EC2 instances, with API, Engines and Web Application listed as affected components. It started at 2021-12-22 12:41 UTC and was resolved at 2021-12-22 20:17 UTC, lasting 7 h 35 min. Firebolt said AWS identified the issue and a fix was in progress before resolution.

Ccrates.ioRequest timeouts/Slow responses

TTwingateAdmin console billing functionality is temporarily unavailable

DDailyConnectivity issuesEvent

MMailjetLogin/Sign up issues

TTwingateInbound requests have heavily downgraded availability

SStream (GetStream)Elevated API Errors on us-east

DDopplerReading/writing secrets intermittently failing due to an outage at our tokenization providerEvent

Doppler reported a major outage causing intermittent failures reading and writing secrets in the API (api.doppler.com) and Dashboard (dashboard.doppler.com). It started at 2021-12-08 02:20 UTC and was resolved at 2021-12-08 10:20 UTC, lasting 8 h 0 min; Doppler attributed it to its tokenization provider being impacted by the AWS us-east-1 outage.

DDopplerReading/writing secrets failing due to an outage at our tokenization providerEvent

Reading and writing secrets failed across Doppler’s API and Dashboard in a major-impact outage from 2021-12-07 18:32 UTC to 2021-12-07 19:01 UTC (28 min). Doppler attributed the failure to its tokenization provider being impacted by the AWS us-east-1 outage; all services were operational by 2021-12-07 18:45 UTC.

DDailyAudio and video calls impacted by an ongoing incident in us-east-1Event

MMetabase Cloudand MetabaseAWS OutageEvent

A major Metabase Cloud Platform outage caused temporary downtime for a few customers due to an AWS us-east-1 issue. It started at 2021-12-07 17:39 UTC and was resolved at 2021-12-07 23:51 UTC, lasting 6 h 12 min; affected customers were operational after AWS restored most services.

MattermostUnable to create new workspacesEvent

Mattermost reported a major outage preventing new workspace creation, affecting Cloud Sign-Up, Cloud Workspaces and Sign-Up, citing a cloud-provider outage in us-east-1. It started at 2021-12-07 16:37 UTC and was resolved at 2021-12-07 22:45 UTC, lasting 6 h 7 min; all systems were reported operational.

FFireboltEngines may fail to start and stopEvent

Firebolt reported a major incident in which engines could fail to start and stop in AWS us-east-1, starting 2021-12-07 15:58 UTC and lasting 15 h 59 min. Firebolt attributed it to partial AWS issues; the incident was resolved at 2021-12-08 07:57 UTC, with engines fully operational.

SStitchExtraction and Loading DelaysEvent

Stitch reported major extraction and loading degradation affecting api.stitchdata.com and app.stitchdata.com, with elevated error rates caused by cloud service provider outages. The incident started at 2021-12-07 15:46 UTC and was resolved at 2021-12-08 01:24 UTC, lasting 9 h 37 min; normal performance and availability were reported at 12-07 19:59 UTC.

FFireboltAccessing Firebolt web application may fail

Firebolt reported a critical web application outage starting 2021-12-07 10:57 UTC, resolved at 2021-12-07 11:00 UTC (3 min). DNS configuration changes could prevent UI access; running workloads and engine operations were unaffected. Firebolt announced a fix rollout and advised users with lingering issues to clear their local DNS cache.

LLogRocketData Processing Delay

LogRocket reported a major data processing delay affecting Alerts, Issues, and Metrics, starting 2021-12-02 19:20 UTC and resolved 2021-12-03 04:13 UTC after 8 h 53 min. The vendor identified an issue with one of its data stores but did not specify a fix.

DDopplerDashboard intermittently returning bad gateway errors

Doppler’s Dashboard (dashboard.doppler.com) intermittently returned bad gateway errors in a major-impact incident starting 2021-11-19 23:48 UTC, resolved at 2021-11-20 00:58 UTC (1 h 9 min). API and CLI usage were unaffected. Doppler suspected connectivity between Cloudflare and its hosting provider and reported identifying and resolving the issue before monitoring the fix.

TTwingateService provider networking outageEvent

DDopplerAll services returning 404

Doppler reported a critical outage with all services returning 404, affecting the API (api.doppler.com) and Dashboard (dashboard.doppler.com). It started at 2021-11-16 17:39 UTC and was resolved at 2021-11-16 18:19 UTC, lasting 39 min. Doppler said it appeared related to a larger Google outage; services were coming back online by 18:09 UTC.