What vendors changed

LLogRocketData processing delays

LogRocket reported major data processing delays affecting Alerts, Issues, and Metrics from 2021-10-05 15:27 UTC to 2021-10-06 18:59 UTC, lasting 27 h 32 min. LogRocket worked with an upstream provider, implemented a fix, and confirmed resolution.

DDopplerAPI unavailable due to contention over database locking

Doppler reported a major API outage caused by database lock contention on its Workplace table. Recorded start: 2021-10-01 19:34 UTC; resolution: 2021-10-01 09:00 UTC; computed duration unknown (vendor reported 11 minutes). Service recovered when locks expired; Doppler refactored the affected component to avoid locking.

JJotformGlobal Elevated Errors

Jotform reported Global Elevated Errors with major impact, starting 2021-09-29 01:00 UTC and resolved at 2021-09-30 01:00 UTC after 24 h 0 min. Customers faced elevated errors; no cause or specific fix was stated.

FFireboltSigning in to Firebolt may fail

Firebolt sign-ins could fail due to an auth0 outage in US-1 and EU regions, affecting the Web Application. Firebolt rated the incident major; it started at 2021-09-28 16:19 UTC and was resolved at 2021-09-28 16:30 UTC, with a reported duration of 10 min.

FFireboltEngines may fail to start and stopEvent

Firebolt reported a major outage in which engines could fail to start and stop due to an issue identified on AWS's side. It started at 2021-09-27 08:30 UTC and was resolved at 2021-09-27 12:13 UTC, lasting 3 h 42 min; Firebolt confirmed it was fully operational.

DDopplerWorkplace admins, viewers and collaborators unable to login via CLI

Doppler reported a major API incident preventing workplace admins, viewers and collaborators from logging in via CLI. It started at 2021-09-23 17:08 UTC and was resolved at 2021-09-23 19:12 UTC, lasting 2 h 4 min. Doppler fixed an erroneous permission check that blocked lower privilege users from authorizing the CLI.

FFireboltSigning in to Firebolt may fail

Firebolt reported a major incident in which sign-ins could fail, affecting AWS ec2-us-east-2, Engines and Web Application. It started at 2021-08-31 18:49 UTC and was resolved at 2021-09-01 02:02 UTC, lasting 7 h 12 min. Firebolt attributed it to cloud-provider networking issues and reported implementing a fix.

FFireboltSigning in to Firebolt may fail

Firebolt reported a major incident in which signing in could fail due to third-party provider authentication issues. It started at 2021-08-19 22:17 UTC and lasted 24 min, ending at 2021-08-19 22:41 UTC with login operational.

MezmoSearch and Live Tail Intermittent Failures

Mezmo reported a major incident affecting Web UI searches and Live Tail from 2021-08-19 13:56 UTC to 2021-08-19 20:48 UTC, lasting 6 h 52 min. Intermittent failures affected some customers; about 12% also experienced delays before newly submitted logs were available for searching, graphing, and timelines. Mezmo took remedial action and reported all services fully operational.

TTwingateController downtime

MMetabase Cloudand MetabaseDegraded service in our Business tier

Metabase Cloud reported major degradation affecting Business tier customers in isolated environments from 2021-08-10 16:37 UTC to 2021-08-10 20:38 UTC, lasting 4 h 1 min. A failed Elastic Beanstalk upgrade left instances inconsistent, with load balancers unable to mark them healthy and application servers continuously recycled; a fix was deployed before resolution.

SStitchData Extraction Delays

Stitch reported a critical data extraction processing outage affecting api.stitchdata.com and app.stitchdata.com, starting 2021-08-07 05:09 UTC and lasting 49 min until 2021-08-07 05:58 UTC. Stitch attributed it to an underlying infrastructure provider outage; a fix was implemented and the incident was resolved.

SSvixDashboard and application portal were temporarily inaccessible for some users

TTwingateController downtime

CodaProblems with loading docs

Coda reported a critical incident affecting doc loading, with API, Doc List, and Docs listed as affected components. It started at 2021-07-26 18:50 UTC and was resolved at 2021-07-26 19:03 UTC; the computed duration was 12 min. No cause or fix was stated.

SStitchGoogle Analytics Extraction Errors

Stitch reported a major Google Analytics outage from 2021-07-20 13:32 UTC to 2021-07-20 17:33 UTC (4 h 1 min), blocking new integrations and extractions. A Google API asset returned 404 errors; Stitch released an integration update to mitigate retrieval failures, then confirmed extraction and configuration success rates were normal and stable.

FFireboltEngines may fail to start and stop

Firebolt reported a major incident in which engines could fail to start or stop, beginning 2021-07-20 04:30 UTC and lasting 4 h 20 min. Firebolt attributed it to unidentified issues on AWS’s side and confirmed resolution there at 2021-07-20 08:50 UTC, restoring engine start/stop functionality.

Travis CIBuilds are not triggering for travis-ci.com

Travis CI reported a major incident preventing some users from triggering travis-ci.com builds on GitHub commits. It started at 2021-07-13 16:41 UTC and was resolved at 2021-07-13 19:45 UTC, lasting 3 h 4 min; the infrastructure team reported implementing a fix at 07-13 17:01 UTC.

LLogRocketSDK breaking lodash on some sites

LogRocket’s major SDK incident broke some customer sites by overwriting lodash’s `window._` global due to a build-pipeline bug. It started and was marked resolved at 2021-07-12 14:20 UTC; computed duration was 0 min, though LogRocket reported about an hour and 40 minutes. LogRocket rolled back to a known functional SDK version.

JJotformGlobal Elevated Errors

Jotform reported “Global Elevated Errors” with critical impact, starting at 2021-06-30 09:28 UTC and marked resolved at the same timestamp; the computed duration was 0 min. Customers may have encountered elevated errors; the update said the issue was solved but gave no cause or fix details.

FFireboltStarting/restarting new engines fails

Firebolt reported a major incident preventing new engines from starting or restarting, beginning 2021-06-29 11:30 UTC and lasting 29 min. A fix was implemented at 2021-06-29 11:55 UTC; engine startup was restored and the incident resolved at 2021-06-29 11:59 UTC.

Travis CISome builds are being rejected

Travis CI reported a critical incident affecting Linux, Windows and Mac build processing, with some build requests rejected. It started at 2021-06-25 07:57 UTC and was resolved at 2021-06-25 08:43 UTC, lasting 45 min; a fix restored normal build-request processing.

Travis CIQueued jobs are taking time to process

Travis CI reported critical delays processing queued builds, affecting Linux, Windows and Mac build processing. The incident started at 2021-06-21 15:05 UTC and resolved at 2021-06-22 08:05 UTC, lasting 16 h 59 min. Travis CI identified the problem and worked on a fix but provided no cause or fix details.

ShopifyAssets not loading for some Shopify services

Shopify reported a critical outage affecting CDN assets and access to Admin, Checkout and Storefront services for some merchants and customers. It started at 2021-06-08 10:24 UTC and was resolved at 2021-06-08 11:20 UTC, lasting 55 min; Shopify identified the problem but did not state a cause or specific fix.

RubyGems.orgElevated ErrorsEvent

LLogRocketSession Ingestion Delays

LogRocket reported major session ingestion delays affecting Alerts and Session Collection from 2021-06-03 15:52 UTC to resolution at 2021-06-04 04:06 UTC, lasting 12 h 13 min. Operations were restored on the primary search cluster and primary session processing returned to real-time; at resolution, secondary processing was still catching up and estimated to be live within 6 hours.

Travis CIPPC64 Jobs Are Failing

Travis CI reported a major incident affecting PPC64 jobs, with slower starts and increased errors in Builds Processing - Linux and Windows Builds and Linux and Windows Builds. It started at 2021-06-01 18:22 UTC and was resolved at 2021-06-01 21:26 UTC, lasting 3 h 3 min; no cause or specific fix was stated.

FFireboltLogging into FIrebolt may fail

Firebolt reported a critical login outage affecting direct and JDBC logins from 2021-05-11 07:04 UTC to 2021-05-11 07:52 UTC, lasting 48 min. Login attempts had high failure rates; customers were advised to retry, while all other functionality remained operational. Firebolt confirmed logins were fully functional at resolution.

SStitchWeb Application Degradation

Stitch reported major degradation affecting connection information in its user interface and API, starting 2021-05-03 20:27 UTC and lasting 1 h 10 min. A fix was deployed at 05-03 20:39 UTC; the incident was resolved at 2021-05-03 21:37 UTC after application and API functionality remained stable during monitoring.

MezmoSearch performance degraded.

Mezmo’s major incident delayed all customers’ logs in the WebUI, Alerting, Live Tail and archive uploads. The postmortem lists the start as April 28, 2021 at 12:37 UTC; resolution was 2021-05-03 00:54 UTC (computed duration: 2 days 12 h). All services were reported fully operational, with archives uploading to their destinations.

SStitchWeb Application Degradation

Stitch reported a major degradation affecting UI access, extraction scheduling and performance, loading performance, and API availability from 2021-04-28 15:52 UTC to 2021-04-28 21:32 UTC (5 h 40 min). Initial errors appeared linked to a spike in internal database errors; a fix was deployed at 17:10 UTC, with services remaining stable through monitoring.

DDopplerSecrets storage is unavailable

Doppler reported a major secrets-storage outage affecting API (api.doppler.com) and Dashboard (dashboard.doppler.com), starting 2021-04-28 03:15 UTC and lasting 11 min until resolution at 2021-04-28 03:26 UTC. Doppler said the incident involved a third-party vendor; a fix was implemented and all secrets access was restored.

CodaProblems with loading docs

Coda reported a critical incident affecting doc loading, starting at 2021-04-27 18:02 UTC and resolved at 2021-04-27 18:18 UTC, with a reported duration of 15 min. Customers could have had trouble loading docs; no cause or fix was stated.

Ccrates.iocrates.io not reachable

MezmoWeb UI is temporarily unavailable

Mezmo reported a major Web UI outage affecting Livetail, Search, and Web App; log searches could return errors. It started at 2021-04-23 06:17 UTC and was resolved at 2021-04-23 06:45 UTC, lasting 27 min. Mezmo reported restoring log access and confirmed the Web UI was fully operational.

JJotformHIPAA - Downtime

Jotform reported critical downtime for HIPAA users, starting at 2021-04-22 12:53 UTC. An application-layer error overloaded the database layer; the vendor marked it fixed at 2021-04-22 11:30 UTC, earlier than the recorded start, so duration is unknown.

LLogRocketAuthentication Service Outage

LogRocket reported a critical Dashboard authentication outage starting 2021-04-20 16:38 UTC, preventing users from logging in because its authentication provider was experiencing an outage. It was marked resolved at 2021-04-20 20:51 UTC after 4 h 13 min; users appeared able to log in again, with monitoring continuing pending the provider’s confirmation of full restoration.

FFireboltLogging into FIrebolt may failEvent

Firebolt reported a critical Web Application incident where login attempts could fail or take significantly longer than usual, starting 2021-04-20 16:25 UTC and resolved at 2021-04-20 17:19 UTC (53 min). Firebolt attributed the issue to an Auth0 outage and reported implementing a fix.

MezmoLogDNA Web UI is unresponsive

Mezmo reported a major incident affecting LogDNA’s Web App, Livetail and Search: pages were not loading consistently. It started at 2021-04-20 09:04 UTC and was resolved at 2021-04-20 11:54 UTC, lasting 2 h 50 min. Engineers applied a fix, and logs were accessible in the web app upon resolution.

Ccrates.iocrates.io unavailable

FFireboltEngines may fail to start or stop

Firebolt reported a major incident in which engines could fail to start or stop, beginning 2021-04-18 10:49 UTC and resolved at 2021-04-18 11:10 UTC, with a reported duration of 20 min. Firebolt said it identified the issue and was implementing a fix, but provided no cause or fix details.

SStream (GetStream)Increased error rate on Chat API

DDopplerCLI Tokens Authentication Issues

Doppler reported a major API (api.doppler.com) incident: CLI token authentication failed with "Invalid CLI Token" errors. It started at 2021-04-14 04:38 UTC and was resolved at 2021-04-14 05:44 UTC, lasting 1 h 5 min. Affected users could work around it by re-logging in with 'doppler login'; Doppler confirmed tokens were functioning again.

MattermostDelay on new cloud workspace signups

Mattermost reported a major delay in new Cloud workspace sign-ups affecting Cloud Sign-Up and Sign-Up from 2021-03-26 09:43 UTC to 2021-03-26 11:32 UTC, lasting 1 h 49 min. The vendor reported identifying and fixing the issue at 03-26 10:39 UTC, then monitoring before resolution; no cause was stated.

Ccrates.iocrates.io unavailable

JJotformHIPAA - Downtime

Jotform reported critical downtime for HIPAA users due to NFS failure and stated that necessary actions were taken. The incident lists a start of 2021-03-18 12:28 UTC and resolution at 2021-03-17 16:30 UTC; duration is unknown, and the resolution timestamp precedes the start.

Travis CISome builds may be affected due to Google Incident

Some Travis CI Linux and Windows build jobs could fail to run due to an ongoing Google Compute Engine incident. Travis CI rated the impact major; it started at 2021-03-17 17:45 UTC and was resolved at 2021-03-17 19:20 UTC, lasting 1 h 34 min.

SStream (GetStream)Chat API

MezmoLogs are not searchable in web app

Mezmo reported a major outage affecting Search, Livetail and the Web App: users could not search logs or use Live Tail in the Web UI. The incident ran from 2021-03-04 08:00 UTC to 2021-03-04 08:21 UTC (21 min); a fix was implemented at 2021-03-04 08:12 UTC.