What vendors changed

Ccrates.iodocs.rs unreachable

Ccrates.iodocs.rs fails to build new releases

Travis CIPartial Outage on Notifications

Travis CI reported a major partial outage affecting Background Processing - Notifications and Notifications, preventing notifications from firing. It started at 2020-04-29 20:56 UTC and lasted 38 min, resolving at 2020-04-29 21:35 UTC. The vendor identified notification infrastructure issues and implemented a fix at 21:03 UTC.

Travis CIBacklog of Jobs on .ORG

Travis CI reported a major backlog of queued MacOS build jobs on .ORG starting at 2020-04-28 22:07 UTC, delaying build processing. A fix was implemented by 2020-04-28 22:25 UTC; the incident was resolved at 2020-04-28 22:57 UTC after 49 min.

Travis CIQueued Jobs on .COM

Travis CI reported a major degradation involving increased queued jobs on .COM, affecting Linux and Windows build processing. It ran from 2020-04-27 20:45 UTC to 2020-04-27 22:32 UTC, lasting 1 h 47 min; a fix was implemented and the incident was resolved.

RubyGems.orgUsers unable to access RubyGems.org

Travis CIDegraded GitHub integrationEvent

Travis CI reported major degradation of its GitHub integration starting 2020-04-21 16:11 UTC, affecting builds, notifications, user sync, and related services due to GitHub's ongoing issues. Service returned to normal at 2020-04-21 16:52 UTC; the recorded duration was 40 min.

Travis CIIntermittent build start or failure issue

Travis CI reported a major incident causing intermittent macOS build start issues or failures, attributed to an infrastructure service provider issue. It started at 2020-04-14 07:10 UTC and was resolved at 2020-04-14 09:45 UTC, lasting 2 h 34 min.

CodaDegraded performance accessing docs

Coda reported a critical-impact Docs incident in which docs failed to load for some users, starting at 2020-04-13 07:27 UTC. A fix was implemented by 2020-04-13 07:41 UTC, and the incident was resolved at 2020-04-13 07:55 UTC after 28 min.

CodaProblems loading, creating docs

Coda reported a major outage affecting doc loading and creation across API, Doc List, and Docs. It started at 2020-04-08 20:58 UTC and was resolved at 2020-04-08 21:51 UTC, lasting 53 min; a fix was implemented and monitored before resolution.

Travis CILinux and Windows VMs not booting

Travis CI reported a major outage preventing Linux and Windows VMs from booting to run builds, starting 2020-04-08 14:57 UTC and lasting 35 min. The vendor attributed it to a service provider issue; at 2020-04-08 15:32 UTC, it reported the issue resolved and said builds should process normally.

Travis CIAPI is down on .ORG

Travis CI’s .ORG API had a critical outage starting 2020-03-27 20:14 UTC, resolved at 2020-03-27 21:51 UTC after 1 h 36 min. Travis CI reported implementing a fix and monitoring its infrastructure and upstream provider Google; customers using the .ORG API were affected.

Travis CIBuilds not starting/queuing or stucks on .COM

Travis CI reported a major .COM outage with builds not starting, queuing or getting stuck, from 2020-03-24 10:42 UTC to 2020-03-24 12:34 UTC, lasting 1 h 51 min. The vendor attributed the problem to “misusage,” reported recovering infrastructure, and confirmed builds should behave normally after resolution.

Travis CIInternal Mac build errors

Travis CI reported a major Mac Builds outage from 2020-03-22 16:23 UTC to 2020-03-22 17:12 UTC, lasting 49 min. A portion of Mac infrastructure prevented builds from running correctly, with affected jobs logging "An error occurred while generating the build script." Travis CI removed affected hardware from the executor pool and restored normal Mac build capacity.

Travis CIMacOS Builds Failing on .COM and .ORG

MacOS builds failed on Travis CI .COM and .ORG in a critical outage from 2020-03-19 19:41 UTC to 2020-03-19 21:59 UTC, lasting 2 h 17 min. Travis CI attributed the failures to a major upstream-provider outage and resolved the incident after the provider reported a fix and Travis CI observed improvements.

Ccrates.ioIncreased errors while connecting to crates.ioEvent

RubyGems.orgUnexpected Problem

DDopplerMigration Script Race Condition

Doppler reported a critical outage starting and resolved at 2020-02-06 23:43 UTC (computed duration: 0 min; vendor reported approximately 2 minutes of downtime). A migration failed to run before deployment, causing services to crash because a critical database table lacked a column; the deployment orchestrator reverted to the previous build.

ShopifyCheckout completion delayed for some stores

Shopify reported a major payment-processing interruption for some stores starting 2020-01-25 20:05 UTC, resolved at 2020-01-25 20:16 UTC after 11 min. Online checkouts were queued, with buyers to be notified upon payment completion; other channels, including Point of Sale (POS), could be unavailable. A fix was implemented at 20:11 UTC.

ShopifyPayPal not processing payments

PayPal payments failed for some Shopify stores in a major-impact incident affecting Checkout and Point of Sale. It started at 2020-01-25 19:48 UTC and was resolved at 2020-01-25 20:13 UTC, lasting 25 min; Shopify reported a PayPal service interruption but did not specify a fix.

SStitchLegacy Database Integration Extraction Issues

Stitch reported a major outage preventing extraction jobs from starting for a subset of legacy database integrations, beginning 2020-01-22 22:26 UTC. The recorded duration was 14 h 12 min, with resolution posted at 2020-01-23 12:39 UTC; Stitch said the issue was resolved circa 06:42 UTC and extraction jobs were running successfully.

ShopifyFailures with Shopify Shipping, Shopify Pay and Admin access

Shopify reported a critical incident involving failures with Shopify Shipping, Shopify Pay and Admin access, affecting Admin and Checkout. It began at 2020-01-20 14:31 UTC and was resolved at 2020-01-20 14:53 UTC, lasting 22 min; Shopify stated that a fix had been implemented and was being monitored.

DDopplerRestricted API Key authentication is down

Doppler reported a critical Restricted API Key authentication outage affecting API (api.doppler.com) and Deploy (deploy.doppler.com), starting 2019-12-09 05:00 UTC and resolved 2019-12-09 07:38 UTC (2 h 38 min). An authentication-revamp logic bug blocked project access with “User must be provided.” Doppler deployed a fix and reported no data loss.

Ccrates.ioIncreased Error Rate

ShopifyAvalara service degradation causing some checkouts for merchants using Avalara to fall back to Shopify tax engine

Shopify reported a major Avalara service degradation that caused some checkouts for merchants using Avalara to fall back to the Shopify tax engine. The incident started at 2019-12-04 22:30 UTC and was resolved at 2019-12-04 23:04 UTC, lasting 33 min; no cause or fix was stated.

ShopifyShopify admin is currently unreachable

Shopify admin was unreachable in a critical incident from 2019-12-03 15:41 UTC to 2019-12-03 15:51 UTC, lasting 9 min. Shopify implemented a fix at 15:49 UTC and confirmed resolution at 15:51 UTC.

ShopifyAdmin and Storefronts experiencing intermittent downtime

Shopify reported major intermittent downtime affecting Admin, Checkout, and Storefront, starting at 2019-11-25 19:46 UTC and lasting 1 h 48 min. A fix was implemented at 20:13 UTC, and the incident was resolved at 2019-11-25 21:34 UTC.

SStream (GetStream)Timeout Errors

ShopifyBuy button malfunctioning in the admin

Shopify reported a major Admin incident in which Buy Buttons were not visible on product pages for some stores. It started at 2019-11-20 23:59 UTC and was resolved at 2019-11-21 00:06 UTC, lasting 7 min; Shopify implemented a fix and monitored the results.

ShopifyShopify storefronts not accessible on some stores

Some Shopify storefronts were inaccessible in a major-impact incident affecting Admin, Checkout, and Storefront. It started at 2019-11-20 18:40 UTC and was resolved at 2019-11-20 18:45 UTC, with a computed duration of 4 min; Shopify did not state a cause or fix.

ShopifyNo packing slips are being served

Shopify’s packing slip service in Admin stopped serving packing slips in a major outage from 2019-11-15 20:21 UTC to 2019-11-15 22:27 UTC, lasting 2 h 6 min. Shopify implemented a fix at 2019-11-15 22:12 UTC and confirmed resolution; no cause was stated.

ShopifyUPS shipping rates not available at checkout

Some Shopify stores could not display UPS shipping rates at checkout; UPS rates from Shopify Shipping were unaffected. The vendor-rated major outage started at 2019-11-14 06:06 UTC, lasted 22 min, and was resolved at 2019-11-14 06:29 UTC. Shopify recommended adding other carriers’ rates so customers could complete orders; no cause or specific fix was stated.

ShopifySome stores unable to purchase UPS shipping labels.

Some Shopify stores could not purchase UPS shipping labels from the fulfillment page in the Shopify admin; checkout shipping options worked normally. Shopify rated the Third party services incident major; it started and was resolved at 2019-11-14 06:05 UTC, with a recorded duration of 0 min. No cause or fix was stated.

ShopifyUPS shipping rates not available at checkout

Some Shopify stores displayed no UPS shipping rates at checkout, potentially preventing customers from completing orders. The vendor rated the incident major; it started at 2019-11-08 04:15 UTC and was resolved at 2019-11-08 07:44 UTC, lasting 3 h 28 min. Shopify recommended adding rates from other carriers while UPS worked on a solution.

ShopifySome stores unable to purchase DHL Express shipping labels.

Some Shopify stores could not purchase DHL Express shipping labels from the fulfillment page in the Shopify admin; customer checkout shipping options were unaffected. Shopify rated the incident major; it started at 2019-11-05 19:29 UTC and was resolved at 2019-11-05 21:32 UTC, lasting 2 h 2 min.

ShopifyStorefront and checkout down for some shops

Shopify reported a major outage affecting Storefront and Checkout for some shops, starting 2019-11-05 16:51 UTC and resolved at 2019-11-05 19:02 UTC, lasting 2 h 10 min. A fix was implemented by 2019-11-05 17:34 UTC; no cause was stated.

ShopifyStorefront and checkouts down for some storefronts

Shopify reported a major outage affecting some storefronts and checkouts, with Admin, Checkout and Storefront listed as affected components. It lasted 5 h 5 min, from 2019-11-01 17:15 UTC to 2019-11-01 22:20 UTC. Shopify implemented a fix and monitored lingering lag and associated issues before resolving the incident.

ShopifyInterruption in service on some stores

Some Shopify stores were inaccessible in a major Storefront outage starting 2019-10-29 16:09 UTC and resolved at 2019-10-29 16:33 UTC, lasting 23 min. Shopify reported implementing a fix but did not state a cause.

ShopifyUPS shipping rates not available at checkout

Some Shopify stores showed no UPS shipping rates at checkout in a major incident from 2019-10-25 14:54 UTC to 2019-10-25 16:10 UTC, lasting 1 h 16 min. Shopify recommended adding other carriers’ rates so customers could complete orders; UPS was working on a solution, and the incident was subsequently marked resolved.

ShopifyPayment processing failures for merchants using PayPal Express

Shopify reported a major Checkout incident: PayPal Express payments failed starting at 2019-10-23 13:48 UTC; other PayPal checkout methods were expected to be unaffected. Payments were processing correctly by 13:57 UTC, and the incident was resolved at 2019-10-23 14:09 UTC, with a computed duration of 20 min.

ShopifyThe Admin is not loading correctly for some stores

Shopify reported a major Admin outage that prevented some merchants from accessing themes, from 2019-10-22 14:09 UTC to 2019-10-22 16:06 UTC, lasting 1 h 57 min. A fix was implemented and the incident was resolved; no cause was stated.

ShopifyStorefront, Admin and Checkout down for some stores

Shopify reported a major outage affecting Storefront, Admin and Checkout for some stores, starting at 2019-10-22 01:03 UTC and resolved at 2019-10-22 01:27 UTC; the computed duration was 23 min. A fix was implemented at 01:25 UTC, but no cause was stated.

ShopifyGoogle Shopping app not syncing

Shopify’s Google Shopping app stopped syncing for some stores in a major-impact incident from 2019-10-17 17:31 UTC to 2019-10-17 18:35 UTC, lasting 1 h 4 min. Shopify implemented a fix at 17:43 UTC and monitored results before marking the incident resolved.

CodaSystem Maintenance

Coda reported major-impact routine maintenance affecting API, Doc List, and Docs, with intermittent service disruption expected. It started at 2019-09-11 03:18 UTC and was resolved at 2019-09-11 04:25 UTC, lasting 1 h 6 min; Coda said maintenance was complete and all services should be operational.

CodaProblems loading, editing docs

Coda reported a critical incident affecting API and Docs, with problems loading and editing docs. It began at 2019-08-13 22:12 UTC and was resolved at 2019-08-13 22:24 UTC, lasting 12 min; no cause or specific fix was stated.

Travis CINetwork Connectivity Errors

Travis CI reported a major incident involving build failures from unsuccessful TLS handshakes to whereami.travis-ci.com:443, affecting Linux, Windows and Mac build processing. It started at 2019-08-08 07:29 UTC and was resolved at 2019-08-08 14:19 UTC, lasting 6 h 50 min. Affected production systems were removed from the pool; all builds were back to normal by 10:52 UTC.

Travis CIUnreliable or intermittent network connectivity for Linux and Windows builds

Travis CI reported a major degradation causing unreliable, intermittent or slow network connectivity for Linux and Windows builds from 2019-08-06 17:55 UTC to 2019-08-07 14:40 UTC, lasting 20 h 45 min. The vendor identified NAT gateway updates as the issue and fully reverted them, restoring normal build network performance and reliability.

CodaIntermittent outage affecting some services

Coda reported a major intermittent outage affecting API and Docs, starting at 2019-07-30 17:37 UTC and resolved at 2019-07-30 18:06 UTC after 29 min. No cause or specific fix was stated.

DDopplerUsers are unable to connect to Doppler servers

Doppler reported a critical outage preventing connections to its API (api.doppler.com), Dashboard (dashboard.doppler.com), and Website (doppler.com), disrupting customer access. It started at 2019-03-06 23:45 UTC and was resolved at 2019-03-07 00:50 UTC, lasting 1 h 5 min. Doppler implemented fixes and reported no data loss.

RubyGems.orgUnexpected Problem