What vendors changed

ShopifyFailures with Shopify Shipping, Shopify Pay and Admin access

Shopify reported a critical incident involving failures with Shopify Shipping, Shopify Pay and Admin access, affecting Admin and Checkout. It began at 2020-01-20 14:31 UTC and was resolved at 2020-01-20 14:53 UTC, lasting 22 min; Shopify stated that a fix had been implemented and was being monitored.

DDopplerRestricted API Key authentication is down

Doppler reported a critical Restricted API Key authentication outage affecting API (api.doppler.com) and Deploy (deploy.doppler.com), starting 2019-12-09 05:00 UTC and resolved 2019-12-09 07:38 UTC (2 h 38 min). An authentication-revamp logic bug blocked project access with “User must be provided.” Doppler deployed a fix and reported no data loss.

Ccrates.ioIncreased Error Rate

ShopifyAvalara service degradation causing some checkouts for merchants using Avalara to fall back to Shopify tax engine

Shopify reported a major Avalara service degradation that caused some checkouts for merchants using Avalara to fall back to the Shopify tax engine. The incident started at 2019-12-04 22:30 UTC and was resolved at 2019-12-04 23:04 UTC, lasting 33 min; no cause or fix was stated.

ShopifyShopify admin is currently unreachable

Shopify admin was unreachable in a critical incident from 2019-12-03 15:41 UTC to 2019-12-03 15:51 UTC, lasting 9 min. Shopify implemented a fix at 15:49 UTC and confirmed resolution at 15:51 UTC.

ShopifyAdmin and Storefronts experiencing intermittent downtime

Shopify reported major intermittent downtime affecting Admin, Checkout, and Storefront, starting at 2019-11-25 19:46 UTC and lasting 1 h 48 min. A fix was implemented at 20:13 UTC, and the incident was resolved at 2019-11-25 21:34 UTC.

SStream (GetStream)Timeout Errors

ShopifyBuy button malfunctioning in the admin

Shopify reported a major Admin incident in which Buy Buttons were not visible on product pages for some stores. It started at 2019-11-20 23:59 UTC and was resolved at 2019-11-21 00:06 UTC, lasting 7 min; Shopify implemented a fix and monitored the results.

ShopifyShopify storefronts not accessible on some stores

Some Shopify storefronts were inaccessible in a major-impact incident affecting Admin, Checkout, and Storefront. It started at 2019-11-20 18:40 UTC and was resolved at 2019-11-20 18:45 UTC, with a computed duration of 4 min; Shopify did not state a cause or fix.

ShopifyNo packing slips are being served

Shopify’s packing slip service in Admin stopped serving packing slips in a major outage from 2019-11-15 20:21 UTC to 2019-11-15 22:27 UTC, lasting 2 h 6 min. Shopify implemented a fix at 2019-11-15 22:12 UTC and confirmed resolution; no cause was stated.

ShopifyUPS shipping rates not available at checkout

Some Shopify stores could not display UPS shipping rates at checkout; UPS rates from Shopify Shipping were unaffected. The vendor-rated major outage started at 2019-11-14 06:06 UTC, lasted 22 min, and was resolved at 2019-11-14 06:29 UTC. Shopify recommended adding other carriers’ rates so customers could complete orders; no cause or specific fix was stated.

ShopifySome stores unable to purchase UPS shipping labels.

Some Shopify stores could not purchase UPS shipping labels from the fulfillment page in the Shopify admin; checkout shipping options worked normally. Shopify rated the Third party services incident major; it started and was resolved at 2019-11-14 06:05 UTC, with a recorded duration of 0 min. No cause or fix was stated.

ShopifyUPS shipping rates not available at checkout

Some Shopify stores displayed no UPS shipping rates at checkout, potentially preventing customers from completing orders. The vendor rated the incident major; it started at 2019-11-08 04:15 UTC and was resolved at 2019-11-08 07:44 UTC, lasting 3 h 28 min. Shopify recommended adding rates from other carriers while UPS worked on a solution.

ShopifySome stores unable to purchase DHL Express shipping labels.

Some Shopify stores could not purchase DHL Express shipping labels from the fulfillment page in the Shopify admin; customer checkout shipping options were unaffected. Shopify rated the incident major; it started at 2019-11-05 19:29 UTC and was resolved at 2019-11-05 21:32 UTC, lasting 2 h 2 min.

ShopifyStorefront and checkout down for some shops

Shopify reported a major outage affecting Storefront and Checkout for some shops, starting 2019-11-05 16:51 UTC and resolved at 2019-11-05 19:02 UTC, lasting 2 h 10 min. A fix was implemented by 2019-11-05 17:34 UTC; no cause was stated.

ShopifyStorefront and checkouts down for some storefronts

Shopify reported a major outage affecting some storefronts and checkouts, with Admin, Checkout and Storefront listed as affected components. It lasted 5 h 5 min, from 2019-11-01 17:15 UTC to 2019-11-01 22:20 UTC. Shopify implemented a fix and monitored lingering lag and associated issues before resolving the incident.

ShopifyInterruption in service on some stores

Some Shopify stores were inaccessible in a major Storefront outage starting 2019-10-29 16:09 UTC and resolved at 2019-10-29 16:33 UTC, lasting 23 min. Shopify reported implementing a fix but did not state a cause.

ShopifyUPS shipping rates not available at checkout

Some Shopify stores showed no UPS shipping rates at checkout in a major incident from 2019-10-25 14:54 UTC to 2019-10-25 16:10 UTC, lasting 1 h 16 min. Shopify recommended adding other carriers’ rates so customers could complete orders; UPS was working on a solution, and the incident was subsequently marked resolved.

ShopifyPayment processing failures for merchants using PayPal Express

Shopify reported a major Checkout incident: PayPal Express payments failed starting at 2019-10-23 13:48 UTC; other PayPal checkout methods were expected to be unaffected. Payments were processing correctly by 13:57 UTC, and the incident was resolved at 2019-10-23 14:09 UTC, with a computed duration of 20 min.

ShopifyThe Admin is not loading correctly for some stores

Shopify reported a major Admin outage that prevented some merchants from accessing themes, from 2019-10-22 14:09 UTC to 2019-10-22 16:06 UTC, lasting 1 h 57 min. A fix was implemented and the incident was resolved; no cause was stated.

ShopifyStorefront, Admin and Checkout down for some stores

Shopify reported a major outage affecting Storefront, Admin and Checkout for some stores, starting at 2019-10-22 01:03 UTC and resolved at 2019-10-22 01:27 UTC; the computed duration was 23 min. A fix was implemented at 01:25 UTC, but no cause was stated.

ShopifyGoogle Shopping app not syncing

Shopify’s Google Shopping app stopped syncing for some stores in a major-impact incident from 2019-10-17 17:31 UTC to 2019-10-17 18:35 UTC, lasting 1 h 4 min. Shopify implemented a fix at 17:43 UTC and monitored results before marking the incident resolved.

CodaSystem Maintenance

Coda reported major-impact routine maintenance affecting API, Doc List, and Docs, with intermittent service disruption expected. It started at 2019-09-11 03:18 UTC and was resolved at 2019-09-11 04:25 UTC, lasting 1 h 6 min; Coda said maintenance was complete and all services should be operational.

CodaProblems loading, editing docs

Coda reported a critical incident affecting API and Docs, with problems loading and editing docs. It began at 2019-08-13 22:12 UTC and was resolved at 2019-08-13 22:24 UTC, lasting 12 min; no cause or specific fix was stated.

Travis CINetwork Connectivity Errors

Travis CI reported a major incident involving build failures from unsuccessful TLS handshakes to whereami.travis-ci.com:443, affecting Linux, Windows and Mac build processing. It started at 2019-08-08 07:29 UTC and was resolved at 2019-08-08 14:19 UTC, lasting 6 h 50 min. Affected production systems were removed from the pool; all builds were back to normal by 10:52 UTC.

Travis CIUnreliable or intermittent network connectivity for Linux and Windows builds

Travis CI reported a major degradation causing unreliable, intermittent or slow network connectivity for Linux and Windows builds from 2019-08-06 17:55 UTC to 2019-08-07 14:40 UTC, lasting 20 h 45 min. The vendor identified NAT gateway updates as the issue and fully reverted them, restoring normal build network performance and reliability.

CodaIntermittent outage affecting some services

Coda reported a major intermittent outage affecting API and Docs, starting at 2019-07-30 17:37 UTC and resolved at 2019-07-30 18:06 UTC after 29 min. No cause or specific fix was stated.

DDopplerUsers are unable to connect to Doppler servers

Doppler reported a critical outage preventing connections to its API (api.doppler.com), Dashboard (dashboard.doppler.com), and Website (doppler.com), disrupting customer access. It started at 2019-03-06 23:45 UTC and was resolved at 2019-03-07 00:50 UTC, lasting 1 h 5 min. Doppler implemented fixes and reported no data loss.

RubyGems.orgUnexpected Problem

RubyGems.orgUnexpected Problem

SStream (GetStream)Elevated API Errors on region EU-WEST

RubyGems.orgElevated Errors

RubyGems.orgElevated Errors

RubyGems.orgUnexpected problems with the Dependencies API.