What vendors changed

RubyGems.orgElevated ErrorsEvent

LLogRocketSession Ingestion Delays

LogRocket reported major session ingestion delays affecting Alerts and Session Collection from 2021-06-03 15:52 UTC to resolution at 2021-06-04 04:06 UTC, lasting 12 h 13 min. Operations were restored on the primary search cluster and primary session processing returned to real-time; at resolution, secondary processing was still catching up and estimated to be live within 6 hours.

Travis CIPPC64 Jobs Are Failing

Travis CI reported a major incident affecting PPC64 jobs, with slower starts and increased errors in Builds Processing - Linux and Windows Builds and Linux and Windows Builds. It started at 2021-06-01 18:22 UTC and was resolved at 2021-06-01 21:26 UTC, lasting 3 h 3 min; no cause or specific fix was stated.

FFireboltLogging into FIrebolt may fail

Firebolt reported a critical login outage affecting direct and JDBC logins from 2021-05-11 07:04 UTC to 2021-05-11 07:52 UTC, lasting 48 min. Login attempts had high failure rates; customers were advised to retry, while all other functionality remained operational. Firebolt confirmed logins were fully functional at resolution.

SStitchWeb Application Degradation

Stitch reported major degradation affecting connection information in its user interface and API, starting 2021-05-03 20:27 UTC and lasting 1 h 10 min. A fix was deployed at 05-03 20:39 UTC; the incident was resolved at 2021-05-03 21:37 UTC after application and API functionality remained stable during monitoring.

MezmoSearch performance degraded.

Mezmo’s major incident delayed all customers’ logs in the WebUI, Alerting, Live Tail and archive uploads. The postmortem lists the start as April 28, 2021 at 12:37 UTC; resolution was 2021-05-03 00:54 UTC (computed duration: 2 days 12 h). All services were reported fully operational, with archives uploading to their destinations.

SStitchWeb Application Degradation

Stitch reported a major degradation affecting UI access, extraction scheduling and performance, loading performance, and API availability from 2021-04-28 15:52 UTC to 2021-04-28 21:32 UTC (5 h 40 min). Initial errors appeared linked to a spike in internal database errors; a fix was deployed at 17:10 UTC, with services remaining stable through monitoring.

DDopplerSecrets storage is unavailable

Doppler reported a major secrets-storage outage affecting API (api.doppler.com) and Dashboard (dashboard.doppler.com), starting 2021-04-28 03:15 UTC and lasting 11 min until resolution at 2021-04-28 03:26 UTC. Doppler said the incident involved a third-party vendor; a fix was implemented and all secrets access was restored.

CodaProblems with loading docs

Coda reported a critical incident affecting doc loading, starting at 2021-04-27 18:02 UTC and resolved at 2021-04-27 18:18 UTC, with a reported duration of 15 min. Customers could have had trouble loading docs; no cause or fix was stated.

Ccrates.iocrates.io not reachable

MezmoWeb UI is temporarily unavailable

Mezmo reported a major Web UI outage affecting Livetail, Search, and Web App; log searches could return errors. It started at 2021-04-23 06:17 UTC and was resolved at 2021-04-23 06:45 UTC, lasting 27 min. Mezmo reported restoring log access and confirmed the Web UI was fully operational.

JJotformHIPAA - Downtime

Jotform reported critical downtime for HIPAA users, starting at 2021-04-22 12:53 UTC. An application-layer error overloaded the database layer; the vendor marked it fixed at 2021-04-22 11:30 UTC, earlier than the recorded start, so duration is unknown.

LLogRocketAuthentication Service Outage

LogRocket reported a critical Dashboard authentication outage starting 2021-04-20 16:38 UTC, preventing users from logging in because its authentication provider was experiencing an outage. It was marked resolved at 2021-04-20 20:51 UTC after 4 h 13 min; users appeared able to log in again, with monitoring continuing pending the provider’s confirmation of full restoration.

FFireboltLogging into FIrebolt may failEvent

Firebolt reported a critical Web Application incident where login attempts could fail or take significantly longer than usual, starting 2021-04-20 16:25 UTC and resolved at 2021-04-20 17:19 UTC (53 min). Firebolt attributed the issue to an Auth0 outage and reported implementing a fix.

MezmoLogDNA Web UI is unresponsive

Mezmo reported a major incident affecting LogDNA’s Web App, Livetail and Search: pages were not loading consistently. It started at 2021-04-20 09:04 UTC and was resolved at 2021-04-20 11:54 UTC, lasting 2 h 50 min. Engineers applied a fix, and logs were accessible in the web app upon resolution.

Ccrates.iocrates.io unavailable

FFireboltEngines may fail to start or stop

Firebolt reported a major incident in which engines could fail to start or stop, beginning 2021-04-18 10:49 UTC and resolved at 2021-04-18 11:10 UTC, with a reported duration of 20 min. Firebolt said it identified the issue and was implementing a fix, but provided no cause or fix details.

SStream (GetStream)Increased error rate on Chat API

DDopplerCLI Tokens Authentication Issues

Doppler reported a major API (api.doppler.com) incident: CLI token authentication failed with "Invalid CLI Token" errors. It started at 2021-04-14 04:38 UTC and was resolved at 2021-04-14 05:44 UTC, lasting 1 h 5 min. Affected users could work around it by re-logging in with 'doppler login'; Doppler confirmed tokens were functioning again.

MattermostDelay on new cloud workspace signups

Mattermost reported a major delay in new Cloud workspace sign-ups affecting Cloud Sign-Up and Sign-Up from 2021-03-26 09:43 UTC to 2021-03-26 11:32 UTC, lasting 1 h 49 min. The vendor reported identifying and fixing the issue at 03-26 10:39 UTC, then monitoring before resolution; no cause was stated.

Ccrates.iocrates.io unavailable

JJotformHIPAA - Downtime

Jotform reported critical downtime for HIPAA users due to NFS failure and stated that necessary actions were taken. The incident lists a start of 2021-03-18 12:28 UTC and resolution at 2021-03-17 16:30 UTC; duration is unknown, and the resolution timestamp precedes the start.

Travis CISome builds may be affected due to Google Incident

Some Travis CI Linux and Windows build jobs could fail to run due to an ongoing Google Compute Engine incident. Travis CI rated the impact major; it started at 2021-03-17 17:45 UTC and was resolved at 2021-03-17 19:20 UTC, lasting 1 h 34 min.

SStream (GetStream)Chat API

MezmoLogs are not searchable in web app

Mezmo reported a major outage affecting Search, Livetail and the Web App: users could not search logs or use Live Tail in the Web UI. The incident ran from 2021-03-04 08:00 UTC to 2021-03-04 08:21 UTC (21 min); a fix was implemented at 2021-03-04 08:12 UTC.

FFireboltWeb application is experiencing a significant outage

Firebolt’s Web Application experienced a critical outage starting at 2021-02-23 09:07 UTC, lasting 6 min and resolving at 2021-02-23 09:14 UTC. Firebolt reported that the issue was identified and a fix was being implemented, but did not specify the cause or fix.

FFireboltEngines may fail to start

Firebolt reported a critical incident in which engines could fail to start, beginning 2021-02-22 17:38 UTC and resolved at 2021-02-22 18:16 UTC, lasting 37 min. The vendor attributed it to a global Terraform/Hashicorp outage and stated that running engines should not be affected.

JJotformHIPAA - Increased HTTP 502 Errors

Jotform reported a major HIPAA-region incident involving increased HTTP 502 errors and storage-layer performance degradation. The recorded start is 2021-02-10 10:30 UTC, but resolution is dated 2021-02-04 16:00 UTC, so duration is unknown; the resolution update states performance returned to normal after 24 minutes of downtime.

JJotformHIPAA - Decryption Service Performance Degradation

Jotform reported critical performance degradation affecting its HIPAA decryption service, starting and resolving at 2021-02-07 16:30 UTC (reported duration: 0 min). The vendor stated that additional resources were added to the zone.

LLogRocketReports of dashboard not loading

LogRocket reported a major Dashboard outage involving reports of the dashboard not loading, starting at 2021-02-05 16:22 UTC and resolved at 2021-02-06 05:14 UTC. The incident lasted 12 h 51 min; no cause or fix was stated.

MezmoWeb UI unavailable and ingestion has stopped

Mezmo reported a critical outage affecting the Web UI and log ingestion, starting at 2021-01-14 20:05 UTC and resolved at 2021-01-14 20:27 UTC (computed duration: 21 min). All services were restored; the postmortem separately reports 45 minutes of disruption and says ingestion client agent logs were automatically resent and successfully ingested.

SStream (GetStream)High error rate on Feed Realtime endpoint

SStream (GetStream)Feed Realtime - SQS high error rate

MezmoWebapp Unavailable

Mezmo reported a major Web App outage affecting https://app.logdna.com from 2020-12-16 22:54 UTC to 2020-12-16 23:02 UTC, lasting 8 min. The vendor identified and resolved the issue but did not specify a cause or fix.

SStream (GetStream)Elevated error rates on Chat API

SStitchInternal Errors Affecting Extraction Jobs and Authentication

Stitch reported a major outage affecting api.stitchdata.com and app.stitchdata.com, causing 401 errors during extraction jobs and integration re-authentication. It started at 2020-10-20 17:15 UTC and was resolved at 2020-10-20 18:27 UTC, lasting 1 h 11 min. Stitch corrected a recent token-change issue that disrupted infrastructure communication; subsequent jobs and authentication attempts succeeded.

CodaGoogle incorrectly marking Coda as a phishing site

Google incorrectly flagged a Coda domain as a phishing site, affecting Doc List and Docs in a major-impact incident starting 2020-09-29 17:24 UTC. Coda contacted Google and worked on mitigation; the incident was resolved at 2020-09-29 18:33 UTC after 1 h 8 min.

Travis CIBuilds are not triggering for travis-ci.com

Travis CI’s major outage prevented travis-ci.com Linux and Windows builds from triggering from 2020-09-24 09:12 UTC to 2020-09-24 10:14 UTC (1 h 1 min). The postmortem cited a Bitbucket implementation infinite loop involving suspended users or missing repository permissions, causing VCS instability and high memory usage; the fix checks the suspended flag.

Ccrates.ioCargo error when downloading crates from our CDN

SStitchDegraded Availability & Service

Stitch reported a major partial billing-service outage affecting account access and functionality on api.stitchdata.com and app.stitchdata.com. It started at 2020-08-24 13:05 UTC and was resolved at 2020-08-24 18:10 UTC, lasting 5 h 4 min. The billing service released a fix, and Stitch reported functionality remained consistent after restoration.

RubyGems.orgElevated DNS Errors

DDopplerDNS resolution failureEvent

Doppler reported a critical DNS outage affecting doppler.com and all subdomains, including API, Dashboard, Deploy, documentation and status-page endpoints, from 2020-07-17 21:20 UTC to 2020-07-17 22:05 UTC (45 min). DNS resolution returned to normal; Doppler initially suspected a Cloudflare outage and reported no data loss.

DDopplerDowntime due to migration script

Doppler reported a critical outage starting 2020-07-10 03:38 UTC, lasting 28 min and resolved at 2020-07-10 04:07 UTC. All primary and failover production servers went down, disabling dashboard and API access; Deploy was also listed as affected, with no data loss reported. The incident title attributed the downtime to a migration script; a fix was implemented.

Ccrates.iodocs.rs outage

Ccrates.iocrates.io outage

Ccrates.iodocs.rs unreachable

Ccrates.iodocs.rs fails to build new releases

Travis CIPartial Outage on Notifications

Travis CI reported a major partial outage affecting Background Processing - Notifications and Notifications, preventing notifications from firing. It started at 2020-04-29 20:56 UTC and lasted 38 min, resolving at 2020-04-29 21:35 UTC. The vendor identified notification infrastructure issues and implemented a fix at 21:03 UTC.

Travis CIBacklog of Jobs on .ORG

Travis CI reported a major backlog of queued MacOS build jobs on .ORG starting at 2020-04-28 22:07 UTC, delaying build processing. A fix was implemented by 2020-04-28 22:25 UTC; the incident was resolved at 2020-04-28 22:57 UTC after 49 min.

Travis CIQueued Jobs on .COM

Travis CI reported a major degradation involving increased queued jobs on .COM, affecting Linux and Windows build processing. It ran from 2020-04-27 20:45 UTC to 2020-04-27 22:32 UTC, lasting 1 h 47 min; a fix was implemented and the incident was resolved.