What vendors changed

DDremioQueries failing in North America AWS regions

Dremio reported a critical incident in North America AWS regions: queries failed to execute while the site remained available. It started at 2022-02-28 18:44 UTC and was resolved at 2022-02-28 19:21 UTC, lasting 36 min. Dremio applied a mitigation and then confirmed a fix was implemented.

MezmoIngestion of new logs — for Syslog only - Partial Outage

Mezmo reported major, Syslog-only log ingestion failures from 2022-02-24 22:45 UTC to resolution at 2022-02-24 23:43 UTC (57 min computed). New logs intermittently failed to ingest. The postmortem gives an earlier start—Thursday, February 18, 2022, at 00:10 UTC—and mentions the newly introduced Syslog Forwarder service.

Ccrates.iocrates.io unavailable

MezmoIngestion of new logs — for Syslog only - is intermittently failing

Mezmo reported a major degradation: new-log ingestion through its Syslog endpoint intermittently failed from 2022-02-18 01:41 UTC to 2022-02-18 02:15 UTC (33 min). The incident was resolved; the postmortem attributed the failures to a code change in Syslog Forwarder, which handles Syslog ingestion.

MezmoSome logs submitted over syslog using custom ports are not being correctly formatted and are not available for Alerting, Searching, Timelines, Graphs, and Live Tail.

Mezmo reported a major Log Ingestion (Syslog) outage affecting some custom-port logs, making them unavailable for Alerting, Searching, Timelines, Graphs, and Live Tail; logs not using custom ports worked normally. It started at 2022-02-16 19:58 UTC and resolved at 2022-02-16 21:56 UTC (1 h 57 min). A fix was implemented.

StatuspagePublic API experiencing increased errors

Statuspage reported a major Public API incident in which increased load caused API errors. The incident started and was resolved at 2022-02-07 18:00 UTC, with a computed duration of 0 min.

MMailjetPartial outbound message delivery delays

LLightstep (ServiceNow Cloud Observability)Errors Ingesting Span Data

LLightstep (ServiceNow Cloud Observability)Delays receiving data from satellites

CodaPacks issues

Coda reported a critical outage affecting Packs starting 2022-01-31 23:15 UTC, resolved at 2022-02-01 00:09 UTC after 54 min. No cause or specific fix was stated.

LLightstep (ServiceNow Cloud Observability)Slack / commands not working

MezmoIngestion, Searching, Live Tail, Alerting, Graphing, and Timelines Delays

Mezmo reported a critical outage: ingestion halted, delaying Searching, Live Tail, Alerting, Graphing, and Timelines; some alerts never triggered. The incident ran from 2022-01-26 16:10 UTC to 2022-01-26 17:15 UTC, lasting 1 h 5 min. A fix restored ingestion and all services by 16:58 UTC.

FFireboltEngines may fail to start and stop

Firebolt reported a major incident affecting engine starts and stops from 2022-01-26 13:02 UTC to 2022-01-26 16:48 UTC (3 h 45 min). Ingestion on impacted engines may also have been affected. Firebolt reported a master-cluster outage that disrupted internal services; a fix was deployed and tested before resolution.

MattermostCollapse threads are not functional in part of Cloud workspaces

Collapsed threads were not functional in some Mattermost Cloud Workspaces during a major-impact incident that started at 2022-01-26 09:05 UTC and was resolved at 2022-01-26 16:04 UTC, lasting 6 h 59 min. Mattermost reported identifying the root cause and working to mitigate affected workspaces, but did not specify the cause or fix.

FFireboltLoading databases could fail

Firebolt reported a critical outage in which database loading could fail, preventing UI connections and potentially affecting API requests. It started at 2022-01-24 09:24 UTC and was resolved at 2022-01-24 13:25 UTC, lasting 4 h 0 min. A fix was implemented and monitored before resolution.

LLightstep (ServiceNow Cloud Observability)Lightstep Webapp Down

TTwingateIssue connecting to Twingate

MMailjetTemplates unavailable for some users

LLightstep (ServiceNow Cloud Observability)Lightstep Webapp and API Experiencing Issues

DDopplerUnable to connect to the Doppler dashboard and API

Doppler reported a critical outage preventing connections to its API (api.doppler.com) and Dashboard (dashboard.doppler.com), starting at 2021-12-22 16:50 UTC. A fix was implemented by 2021-12-22 16:54 UTC, and the incident was resolved at 2021-12-22 17:08 UTC; the computed duration was 17 min.

FFireboltAmazon Elastic Compute outageEvent

Firebolt reported a major Amazon Elastic Compute outage affecting AWS EC2 instances, with API, Engines and Web Application listed as affected components. It started at 2021-12-22 12:41 UTC and was resolved at 2021-12-22 20:17 UTC, lasting 7 h 35 min. Firebolt said AWS identified the issue and a fix was in progress before resolution.

Ccrates.ioRequest timeouts/Slow responses

TTwingateAdmin console billing functionality is temporarily unavailable

DDailyConnectivity issuesEvent

MMailjetLogin/Sign up issues

TTwingateInbound requests have heavily downgraded availability

SStream (GetStream)Elevated API Errors on us-east

DDopplerReading/writing secrets intermittently failing due to an outage at our tokenization providerEvent

Doppler reported a major outage causing intermittent failures reading and writing secrets in the API (api.doppler.com) and Dashboard (dashboard.doppler.com). It started at 2021-12-08 02:20 UTC and was resolved at 2021-12-08 10:20 UTC, lasting 8 h 0 min; Doppler attributed it to its tokenization provider being impacted by the AWS us-east-1 outage.

DDopplerReading/writing secrets failing due to an outage at our tokenization providerEvent

Reading and writing secrets failed across Doppler’s API and Dashboard in a major-impact outage from 2021-12-07 18:32 UTC to 2021-12-07 19:01 UTC (28 min). Doppler attributed the failure to its tokenization provider being impacted by the AWS us-east-1 outage; all services were operational by 2021-12-07 18:45 UTC.

DDailyAudio and video calls impacted by an ongoing incident in us-east-1Event

MMetabase Cloudand MetabaseAWS OutageEvent

A major Metabase Cloud Platform outage caused temporary downtime for a few customers due to an AWS us-east-1 issue. It started at 2021-12-07 17:39 UTC and was resolved at 2021-12-07 23:51 UTC, lasting 6 h 12 min; affected customers were operational after AWS restored most services.

MattermostUnable to create new workspacesEvent

Mattermost reported a major outage preventing new workspace creation, affecting Cloud Sign-Up, Cloud Workspaces and Sign-Up, citing a cloud-provider outage in us-east-1. It started at 2021-12-07 16:37 UTC and was resolved at 2021-12-07 22:45 UTC, lasting 6 h 7 min; all systems were reported operational.

FFireboltEngines may fail to start and stopEvent

Firebolt reported a major incident in which engines could fail to start and stop in AWS us-east-1, starting 2021-12-07 15:58 UTC and lasting 15 h 59 min. Firebolt attributed it to partial AWS issues; the incident was resolved at 2021-12-08 07:57 UTC, with engines fully operational.

SStitchExtraction and Loading DelaysEvent

Stitch reported major extraction and loading degradation affecting api.stitchdata.com and app.stitchdata.com, with elevated error rates caused by cloud service provider outages. The incident started at 2021-12-07 15:46 UTC and was resolved at 2021-12-08 01:24 UTC, lasting 9 h 37 min; normal performance and availability were reported at 12-07 19:59 UTC.

FFireboltAccessing Firebolt web application may fail

Firebolt reported a critical web application outage starting 2021-12-07 10:57 UTC, resolved at 2021-12-07 11:00 UTC (3 min). DNS configuration changes could prevent UI access; running workloads and engine operations were unaffected. Firebolt announced a fix rollout and advised users with lingering issues to clear their local DNS cache.

LLogRocketData Processing Delay

LogRocket reported a major data processing delay affecting Alerts, Issues, and Metrics, starting 2021-12-02 19:20 UTC and resolved 2021-12-03 04:13 UTC after 8 h 53 min. The vendor identified an issue with one of its data stores but did not specify a fix.

DDopplerDashboard intermittently returning bad gateway errors

Doppler’s Dashboard (dashboard.doppler.com) intermittently returned bad gateway errors in a major-impact incident starting 2021-11-19 23:48 UTC, resolved at 2021-11-20 00:58 UTC (1 h 9 min). API and CLI usage were unaffected. Doppler suspected connectivity between Cloudflare and its hosting provider and reported identifying and resolving the issue before monitoring the fix.

TTwingateService provider networking outageEvent

DDopplerAll services returning 404

Doppler reported a critical outage with all services returning 404, affecting the API (api.doppler.com) and Dashboard (dashboard.doppler.com). It started at 2021-11-16 17:39 UTC and was resolved at 2021-11-16 18:19 UTC, lasting 39 min. Doppler said it appeared related to a larger Google outage; services were coming back online by 18:09 UTC.

DDopplerUsers getting rate limited when hitting API

Doppler reported a major API (api.doppler.com) degradation affecting Service Tokens, which were being rate limited. The incident started at 2021-11-15 22:14 UTC and lasted 27 min, resolving at 2021-11-15 22:41 UTC after Doppler rolled back the code change causing the issue.

FFireboltUsing Firebolt web application may fail

Firebolt reported a critical incident in which its web application could fail for customers in the European region. It started at 2021-11-10 10:17 UTC and was resolved at 2021-11-10 10:23 UTC, lasting 6 min; Firebolt reported a fix in progress before resolution but gave no cause.

SStitchWebhooks errors

Stitch’s webhooks receiver API (api.stitchdata.com) experienced elevated error rates with major impact from 2021-11-05 13:58 UTC to 2021-11-05 14:34 UTC, lasting 35 min. Stitch reported identifying the root cause and releasing a fix, then marked the issue resolved.

MezmoEmails Are Not Being Sent

Mezmo reported a major outage affecting US email alerts and delaying email notifications from 2021-10-28 18:13 UTC to 2021-10-28 23:22 UTC (5 h 9 min). Mezmo attributed it to an incident at email provider Sparkpost; Slack and webhook notifications were unaffected. Email alerting was restored.

SStitchApp Page Loading and Job Scheduling

Stitch reported a major outage affecting api.stitchdata.com and app.stitchdata.com, causing page load failures and job scheduling delays from 2021-10-08 13:13 UTC to 2021-10-08 18:24 UTC, lasting 5 h 10 min. The vendor attributed the issue to a cache repopulation process imposing heavy system load; a fix was implemented and the incident resolved.

MezmoSome customers are not able to login to our UI

Mezmo reported a major Web App outage preventing some customers from logging in or loading pages from 2021-10-07 17:52 UTC to 2021-10-07 18:46 UTC (53 min). Log ingestion was unaffected; the postmortem attributed the disruption to a major network routing outage at Telia in Europe. All services were operational at resolution.

MezmoEmail Alerts Are Not Being Sent

Mezmo reported a major email-alerting outage starting 2021-10-06 21:29 UTC, lasting 1 h 5 min and resolving at 2021-10-06 22:35 UTC; Slack and webhook alerts remained functional. The postmortem attributed partial email-notification disruption to exceeding its email provider’s daily limit of 250,000 messages; email alerting was restored.

LLogRocketData processing delays

LogRocket reported major data processing delays affecting Alerts, Issues, and Metrics from 2021-10-05 15:27 UTC to 2021-10-06 18:59 UTC, lasting 27 h 32 min. LogRocket worked with an upstream provider, implemented a fix, and confirmed resolution.

DDopplerAPI unavailable due to contention over database locking

Doppler reported a major API outage caused by database lock contention on its Workplace table. Recorded start: 2021-10-01 19:34 UTC; resolution: 2021-10-01 09:00 UTC; computed duration unknown (vendor reported 11 minutes). Service recovered when locks expired; Doppler refactored the affected component to avoid locking.

JJotformGlobal Elevated Errors

Jotform reported Global Elevated Errors with major impact, starting 2021-09-29 01:00 UTC and resolved at 2021-09-30 01:00 UTC after 24 h 0 min. Customers faced elevated errors; no cause or specific fix was stated.