What vendors changed

MattermostDelay on new cloud workspace signups

Mattermost reported a major delay in new Cloud workspace sign-ups affecting Cloud Sign-Up and Sign-Up from 2021-03-26 09:43 UTC to 2021-03-26 11:32 UTC, lasting 1 h 49 min. The vendor reported identifying and fixing the issue at 03-26 10:39 UTC, then monitoring before resolution; no cause was stated.

Ccrates.iocrates.io unavailable

JJotformHIPAA - Downtime

Jotform reported critical downtime for HIPAA users due to NFS failure and stated that necessary actions were taken. The incident lists a start of 2021-03-18 12:28 UTC and resolution at 2021-03-17 16:30 UTC; duration is unknown, and the resolution timestamp precedes the start.

Travis CISome builds may be affected due to Google Incident

Some Travis CI Linux and Windows build jobs could fail to run due to an ongoing Google Compute Engine incident. Travis CI rated the impact major; it started at 2021-03-17 17:45 UTC and was resolved at 2021-03-17 19:20 UTC, lasting 1 h 34 min.

SStream (GetStream)Chat API

MezmoLogs are not searchable in web app

Mezmo reported a major outage affecting Search, Livetail and the Web App: users could not search logs or use Live Tail in the Web UI. The incident ran from 2021-03-04 08:00 UTC to 2021-03-04 08:21 UTC (21 min); a fix was implemented at 2021-03-04 08:12 UTC.

FFireboltWeb application is experiencing a significant outage

Firebolt’s Web Application experienced a critical outage starting at 2021-02-23 09:07 UTC, lasting 6 min and resolving at 2021-02-23 09:14 UTC. Firebolt reported that the issue was identified and a fix was being implemented, but did not specify the cause or fix.

FFireboltEngines may fail to start

Firebolt reported a critical incident in which engines could fail to start, beginning 2021-02-22 17:38 UTC and resolved at 2021-02-22 18:16 UTC, lasting 37 min. The vendor attributed it to a global Terraform/Hashicorp outage and stated that running engines should not be affected.

JJotformHIPAA - Increased HTTP 502 Errors

Jotform reported a major HIPAA-region incident involving increased HTTP 502 errors and storage-layer performance degradation. The recorded start is 2021-02-10 10:30 UTC, but resolution is dated 2021-02-04 16:00 UTC, so duration is unknown; the resolution update states performance returned to normal after 24 minutes of downtime.

JJotformHIPAA - Decryption Service Performance Degradation

Jotform reported critical performance degradation affecting its HIPAA decryption service, starting and resolving at 2021-02-07 16:30 UTC (reported duration: 0 min). The vendor stated that additional resources were added to the zone.

LLogRocketReports of dashboard not loading

LogRocket reported a major Dashboard outage involving reports of the dashboard not loading, starting at 2021-02-05 16:22 UTC and resolved at 2021-02-06 05:14 UTC. The incident lasted 12 h 51 min; no cause or fix was stated.

MezmoWeb UI unavailable and ingestion has stopped

Mezmo reported a critical outage affecting the Web UI and log ingestion, starting at 2021-01-14 20:05 UTC and resolved at 2021-01-14 20:27 UTC (computed duration: 21 min). All services were restored; the postmortem separately reports 45 minutes of disruption and says ingestion client agent logs were automatically resent and successfully ingested.

SStream (GetStream)High error rate on Feed Realtime endpoint

SStream (GetStream)Feed Realtime - SQS high error rate

MezmoWebapp Unavailable

Mezmo reported a major Web App outage affecting https://app.logdna.com from 2020-12-16 22:54 UTC to 2020-12-16 23:02 UTC, lasting 8 min. The vendor identified and resolved the issue but did not specify a cause or fix.

SStream (GetStream)Elevated error rates on Chat API

SStitchInternal Errors Affecting Extraction Jobs and Authentication

Stitch reported a major outage affecting api.stitchdata.com and app.stitchdata.com, causing 401 errors during extraction jobs and integration re-authentication. It started at 2020-10-20 17:15 UTC and was resolved at 2020-10-20 18:27 UTC, lasting 1 h 11 min. Stitch corrected a recent token-change issue that disrupted infrastructure communication; subsequent jobs and authentication attempts succeeded.

CodaGoogle incorrectly marking Coda as a phishing site

Google incorrectly flagged a Coda domain as a phishing site, affecting Doc List and Docs in a major-impact incident starting 2020-09-29 17:24 UTC. Coda contacted Google and worked on mitigation; the incident was resolved at 2020-09-29 18:33 UTC after 1 h 8 min.

Travis CIBuilds are not triggering for travis-ci.com

Travis CI’s major outage prevented travis-ci.com Linux and Windows builds from triggering from 2020-09-24 09:12 UTC to 2020-09-24 10:14 UTC (1 h 1 min). The postmortem cited a Bitbucket implementation infinite loop involving suspended users or missing repository permissions, causing VCS instability and high memory usage; the fix checks the suspended flag.

Ccrates.ioCargo error when downloading crates from our CDN

SStitchDegraded Availability & Service

Stitch reported a major partial billing-service outage affecting account access and functionality on api.stitchdata.com and app.stitchdata.com. It started at 2020-08-24 13:05 UTC and was resolved at 2020-08-24 18:10 UTC, lasting 5 h 4 min. The billing service released a fix, and Stitch reported functionality remained consistent after restoration.

RubyGems.orgElevated DNS Errors

DDopplerDNS resolution failureEvent

Doppler reported a critical DNS outage affecting doppler.com and all subdomains, including API, Dashboard, Deploy, documentation and status-page endpoints, from 2020-07-17 21:20 UTC to 2020-07-17 22:05 UTC (45 min). DNS resolution returned to normal; Doppler initially suspected a Cloudflare outage and reported no data loss.

DDopplerDowntime due to migration script

Doppler reported a critical outage starting 2020-07-10 03:38 UTC, lasting 28 min and resolved at 2020-07-10 04:07 UTC. All primary and failover production servers went down, disabling dashboard and API access; Deploy was also listed as affected, with no data loss reported. The incident title attributed the downtime to a migration script; a fix was implemented.

Ccrates.iodocs.rs outage

Ccrates.iocrates.io outage

Ccrates.iodocs.rs unreachable

Ccrates.iodocs.rs fails to build new releases

Travis CIPartial Outage on Notifications

Travis CI reported a major partial outage affecting Background Processing - Notifications and Notifications, preventing notifications from firing. It started at 2020-04-29 20:56 UTC and lasted 38 min, resolving at 2020-04-29 21:35 UTC. The vendor identified notification infrastructure issues and implemented a fix at 21:03 UTC.

Travis CIBacklog of Jobs on .ORG

Travis CI reported a major backlog of queued MacOS build jobs on .ORG starting at 2020-04-28 22:07 UTC, delaying build processing. A fix was implemented by 2020-04-28 22:25 UTC; the incident was resolved at 2020-04-28 22:57 UTC after 49 min.

Travis CIQueued Jobs on .COM

Travis CI reported a major degradation involving increased queued jobs on .COM, affecting Linux and Windows build processing. It ran from 2020-04-27 20:45 UTC to 2020-04-27 22:32 UTC, lasting 1 h 47 min; a fix was implemented and the incident was resolved.

RubyGems.orgUsers unable to access RubyGems.org

Travis CIDegraded GitHub integrationEvent

Travis CI reported major degradation of its GitHub integration starting 2020-04-21 16:11 UTC, affecting builds, notifications, user sync, and related services due to GitHub's ongoing issues. Service returned to normal at 2020-04-21 16:52 UTC; the recorded duration was 40 min.

Travis CIIntermittent build start or failure issue

Travis CI reported a major incident causing intermittent macOS build start issues or failures, attributed to an infrastructure service provider issue. It started at 2020-04-14 07:10 UTC and was resolved at 2020-04-14 09:45 UTC, lasting 2 h 34 min.

CodaDegraded performance accessing docs

Coda reported a critical-impact Docs incident in which docs failed to load for some users, starting at 2020-04-13 07:27 UTC. A fix was implemented by 2020-04-13 07:41 UTC, and the incident was resolved at 2020-04-13 07:55 UTC after 28 min.

CodaProblems loading, creating docs

Coda reported a major outage affecting doc loading and creation across API, Doc List, and Docs. It started at 2020-04-08 20:58 UTC and was resolved at 2020-04-08 21:51 UTC, lasting 53 min; a fix was implemented and monitored before resolution.

Travis CILinux and Windows VMs not booting

Travis CI reported a major outage preventing Linux and Windows VMs from booting to run builds, starting 2020-04-08 14:57 UTC and lasting 35 min. The vendor attributed it to a service provider issue; at 2020-04-08 15:32 UTC, it reported the issue resolved and said builds should process normally.

Travis CIAPI is down on .ORG

Travis CI’s .ORG API had a critical outage starting 2020-03-27 20:14 UTC, resolved at 2020-03-27 21:51 UTC after 1 h 36 min. Travis CI reported implementing a fix and monitoring its infrastructure and upstream provider Google; customers using the .ORG API were affected.

Travis CIBuilds not starting/queuing or stucks on .COM

Travis CI reported a major .COM outage with builds not starting, queuing or getting stuck, from 2020-03-24 10:42 UTC to 2020-03-24 12:34 UTC, lasting 1 h 51 min. The vendor attributed the problem to “misusage,” reported recovering infrastructure, and confirmed builds should behave normally after resolution.

Travis CIInternal Mac build errors

Travis CI reported a major Mac Builds outage from 2020-03-22 16:23 UTC to 2020-03-22 17:12 UTC, lasting 49 min. A portion of Mac infrastructure prevented builds from running correctly, with affected jobs logging "An error occurred while generating the build script." Travis CI removed affected hardware from the executor pool and restored normal Mac build capacity.

Travis CIMacOS Builds Failing on .COM and .ORG

MacOS builds failed on Travis CI .COM and .ORG in a critical outage from 2020-03-19 19:41 UTC to 2020-03-19 21:59 UTC, lasting 2 h 17 min. Travis CI attributed the failures to a major upstream-provider outage and resolved the incident after the provider reported a fix and Travis CI observed improvements.

Ccrates.ioIncreased errors while connecting to crates.ioEvent

RubyGems.orgUnexpected Problem

DDopplerMigration Script Race Condition

Doppler reported a critical outage starting and resolved at 2020-02-06 23:43 UTC (computed duration: 0 min; vendor reported approximately 2 minutes of downtime). A migration failed to run before deployment, causing services to crash because a critical database table lacked a column; the deployment orchestrator reverted to the previous build.

ShopifyCheckout completion delayed for some stores

Shopify reported a major payment-processing interruption for some stores starting 2020-01-25 20:05 UTC, resolved at 2020-01-25 20:16 UTC after 11 min. Online checkouts were queued, with buyers to be notified upon payment completion; other channels, including Point of Sale (POS), could be unavailable. A fix was implemented at 20:11 UTC.

ShopifyPayPal not processing payments

PayPal payments failed for some Shopify stores in a major-impact incident affecting Checkout and Point of Sale. It started at 2020-01-25 19:48 UTC and was resolved at 2020-01-25 20:13 UTC, lasting 25 min; Shopify reported a PayPal service interruption but did not specify a fix.

SStitchLegacy Database Integration Extraction Issues

Stitch reported a major outage preventing extraction jobs from starting for a subset of legacy database integrations, beginning 2020-01-22 22:26 UTC. The recorded duration was 14 h 12 min, with resolution posted at 2020-01-23 12:39 UTC; Stitch said the issue was resolved circa 06:42 UTC and extraction jobs were running successfully.

ShopifyFailures with Shopify Shipping, Shopify Pay and Admin access

Shopify reported a critical incident involving failures with Shopify Shipping, Shopify Pay and Admin access, affecting Admin and Checkout. It began at 2020-01-20 14:31 UTC and was resolved at 2020-01-20 14:53 UTC, lasting 22 min; Shopify stated that a fix had been implemented and was being monitored.

DDopplerRestricted API Key authentication is down

Doppler reported a critical Restricted API Key authentication outage affecting API (api.doppler.com) and Deploy (deploy.doppler.com), starting 2019-12-09 05:00 UTC and resolved 2019-12-09 07:38 UTC (2 h 38 min). An authentication-revamp logic bug blocked project access with “User must be provided.” Doppler deployed a fix and reported no data loss.

Ccrates.ioIncreased Error Rate