What vendors changed

PPipedreamv1 email sources not emitting events

Email sources tied to Pipedream v1 workflows failed to emit events, affecting email-triggered workflows. The vendor rated the incident major; it started at 2024-01-06 21:00 UTC and was resolved at 2024-01-06 23:31 UTC, with a reported duration of 2 h 30 min. No cause or fix was stated.

PulumiResource search timeouts

Pulumi reported a major incident in which resource search timed out, affecting API, Console, Insights, and Search. It started at 2024-01-05 20:00 UTC and was resolved at 2024-01-05 21:37 UTC, lasting 1 h 36 min. Pulumi said it identified the issue and search should be functioning normally; no specific cause or fix was provided.

GustoWe are currently experiencing higher than normal call volume. During this time, please leverage help center for support.

Gusto reported higher-than-normal call volume affecting its Phone System, rated critical, and directed customers to its help center for support. The incident started at 2024-01-03 19:42 UTC and was resolved at 2024-01-03 22:00 UTC, lasting 2 h 17 min; no cause or fix was stated.

KeeperVault login errors (Resolved)

Keeper reported critical Vault login errors, starting and resolved at 2024-01-01 15:30 UTC, with a recorded duration of 0 min. Keeper stated that AWS RDS cluster instances automatically rebooted at approximately 04:27 UTC and that it was investigating with AWS; no fix was stated.

Ccrates.ioFailing requests to crates.io

CockroachDBFailures on cluster operations

CockroachDB reported a major incident affecting AWS cluster operations, including creation and modification, starting 2023-12-22 02:01 UTC and lasting 24 h 45 min. Impact later narrowed to vertical scaling failures and delays. Resolved at 2023-12-23 02:46 UTC, the vendor stated all cluster operations should succeed and monitoring continued.

PPandaDocCan't create, view and edit documents

PandaDoc reported a critical outage preventing document creation, viewing and editing, with API, integrations and other US & Global components affected. It started at 2023-12-21 12:58 UTC and was resolved at 2023-12-21 19:00 UTC, lasting 6 h 1 min; PandaDoc reported normal application use restored during monitoring but disclosed no cause or specific fix.

PPandaDocCan't create documents

PandaDoc reported a critical outage preventing document creation, affecting Creating and editing documents and US & Global - Creating and editing documents. It started at 2023-12-20 11:50 UTC and was resolved at 2023-12-20 12:06 UTC; the computed duration was 15 min. PandaDoc reported implementing a fix and confirmed normal operation.

GGrainZoom join failures

Grain reported a critical outage starting 2023-12-19 15:33 UTC: Zoom blocked Grain Recorder from joining approximately 75% of calls. A temporary mitigation appeared successful; Grain later worked with Zoom to deploy a proper fix, reporting no further issues. The incident was resolved 2024-01-19 17:40 UTC, with a recorded duration of 31 days 2 h.

KeeperVault Login in GovCloud (US_GOV) Region

Keeper reported a critical incident affecting vault login in GovCloud (US_GOV), starting at 2023-12-19 00:03 UTC and resolved at 2023-12-19 00:14 UTC, lasting 11 min. The vendor linked login errors to a recent backend app update and restored services by reverting it in GovCloud.

PPipedreamHTTP Sources Outage

Pipedream reported a critical outage involving HTTP source timeouts, with temporary impact to its website, public API and platform; timer-based sources and workflows were later degraded by an event backlog. The incident started at 2023-12-14 03:05 UTC and was resolved at 2023-12-14 03:28 UTC, lasting 22 min; Pipedream reported mitigation but did not specify the root cause.

KeeperDuo Push in US region

Keeper reported major intermittent Duo Push issues in the US region affecting Keeper Browser Extensions and Keeper Web Vault (US), starting 2023-12-13 17:29 UTC and resolved at 2023-12-13 18:19 UTC (49 min). Hard refreshing appeared to mitigate the issue; after resolution, customers might need to reload the web vault or restart their browser.

OpsgenieIOS based Alert Notifications delivered as non-critical

Some iOS alert notifications were delivered as non-critical, affecting EU and US mobile notification delivery with major impact. The incident started at 2023-12-13 13:18 UTC and was resolved at 2023-12-13 14:37 UTC, lasting 1 h 19 min; Opsgenie reported implementing a fix.

DDropbox SignGet Signatures is not working as expected.

Dropbox Sign reported a major outage affecting Get Signatures and the Core - Send signature requests and Send signature requests components. It started at 2023-12-08 17:51 UTC and was resolved at 2023-12-08 21:51 UTC, lasting 3 h 59 min; no cause or specific fix was stated.

ZZillizPrivate Link Connection Issues

PusherPublic EU cluster outage

Pusher reported a critical public EU cluster outage affecting customers in that region; other regions were unaffected. The incident ran from 2023-12-05 17:34 UTC to 2023-12-05 18:11 UTC, lasting 36 min. Pusher identified the issue and implemented a fix, reporting all services back online at 17:47 UTC.

FFlagsmithService Outage

PulumiDeployments stuck in running state

Pulumi reported a major Deployments incident affecting a subset of deployments stuck in the running state. It started at 2023-11-30 22:16 UTC and was resolved at 2023-11-30 23:59 UTC, lasting 1 h 42 min. Pulumi identified the root cause without disclosing details and confirmed all deployments should be functioning normally.

TrelloTrello is unavailable

Trello reported a critical outage affecting API and Trello.com, with errors across web, iOS and Android apps, starting 2023-11-30 14:21 UTC and lasting 3 h 16 min until resolution at 2023-11-30 17:37 UTC. The postmortem attributed it to a code change that overloaded a critical database component; Trello confirmed its database fix restored service.

GGrainAudio corruption on some recordings

Grain Recorder experienced audio “popping” on some calls in a critical-impact incident starting 2023-11-28 18:34 UTC and lasting 20 days 23 h. A fix was deployed by 11-29 15:37 UTC, but affected recordings cannot be repaired. The incident was resolved at 2023-12-19 18:20 UTC, with no further instances reported since the fix.

GustoSome users are unable to access or log into the Gusto website

Some users could not access or log into the Gusto.com website during a critical-impact outage that started at 2023-11-27 18:45 UTC and was resolved at 2023-11-27 19:43 UTC, lasting 58 min. Gusto did not state a cause or fix.

GustoSome users are unable to access the Gusto website, mobile app, and API

Some users could not access Gusto’s website, mobile app, and API during a critical outage that started at 2023-11-27 14:42 UTC and was resolved at 2023-11-27 15:48 UTC, lasting 1 h 6 min. Gusto did not state a cause or specific fix.

CockroachDBSeverless clusters in GCP europe-west1 inaccessible

CockroachDB reported a major outage that made Serverless clusters in GCP europe-west1 inaccessible. The recorded start was 2023-11-21 14:14 UTC, but the resolved timestamp was earlier, at 2023-11-21 12:00 UTC; duration is unknown. No cause or fix was stated.

KeeperFIDO2 / Yubikey issues

Keeper reported a major incident affecting FIDO2 / Yubikey logins in Keeper Web Vault (US), starting 2023-11-17 17:08 UTC and resolved at 2023-11-17 17:31 UTC; the computed duration was 22 min. The engineering team was still investigating the root cause when resolution was posted.

StatuspageIntermittent issues in incidents shown in Status embed frame

Statuspage reported a major incident affecting the Status Embed Widget: a few customers intermittently saw incorrect cached responses in their embedded status frames. It started at 2023-11-17 16:37 UTC and was resolved at 2023-11-17 17:25 UTC, lasting 48 min.

TrelloDegraded experience for some of Trello's users

Trello.com had a major degradation affecting some users: Automation commands failed to run or could not be created, card changes did not save, and some file uploads failed. It started at 2023-11-16 22:37 UTC and was resolved at 2023-11-16 23:58 UTC, lasting 1 h 20 min. Engineers rolled out a fix and observed recovery.

Ccrates.ioFaulty deployment

TrelloTrello is down

Trello reported a critical outage affecting API and Trello.com, preventing multiple users from connecting to its servers. It started at 2023-11-10 14:43 UTC and was resolved at 2023-11-10 16:19 UTC, lasting 1 h 35 min. Trello said its changes fixed the connection issues and all services were functional.

MapboxUploads API jobs not processing

Mapbox reported a major Uploads API degradation: jobs were not processing from 2023-11-10 00:23 UTC until resolution at 2023-11-10 02:18 UTC, lasting 1 h 54 min. Newly created uploads were processing again by 01:02 UTC; at resolution, operations were normal and the backlog was cleared.

Metabaseand MMetabase CloudIssue with SSL certificates

Metabase Cloud Platform had a critical SSL certificate outage from 2023-11-09 19:37 UTC to 2023-11-09 20:02 UTC (24 min), causing invalid certificate errors for users accessing hosted instances. A Helm chart release CI misconfiguration replaced the external Nginx ingress certificate with a dummy certificate; ingress certificates were restored.

SSigma ComputingSigma organizations hosted on AWS US are not loading

A critical outage left Sigma organizations hosted on AWS US stuck on loading screens, with impact starting approximately 20:39 UTC November 08, 2023. The incident record ran from 2023-11-08 20:53 UTC to resolution at 2023-11-08 21:54 UTC (computed duration: 1 h 0 min); mitigation measures restored app loading.

KeeperVault Login Errors

Keeper reported critical vault login errors affecting Keeper Web Vault in AU, EU and US, starting 2023-11-06 09:15 UTC and resolved at 2023-11-06 09:24 UTC (computed duration: 8 min). The postmortem attributed the login errors to an AWS Elasticache failure related to an earlier AWS Aurora database failover.

PPipedreamTimer-based triggers not running

Timer-based triggers, including Scheduler and scheduled app triggers, stopped around 21:30 UTC, affecting v1 workflows but not v2. Pipedream classified the outage as major, logged its start at 2023-11-03 21:49 UTC, and resolved it at 2023-11-03 22:00 UTC (computed duration: 10 min) after reverting code that introduced the bug.

PPipedreamGitHub incident affecting Pipedream GitHub sync

Pipedream reported a critical incident affecting its GitHub integration and GitHub Sync, attributed to a GitHub incident. It started at 2023-11-03 19:06 UTC and was resolved at 2023-11-03 19:18 UTC, with a reported duration of 11 min.

CodaSite wide issue

Coda reported a major site-wide outage affecting access to API, Doc List, Docs, Packs, and Published Docs. It started at 2023-11-03 02:50 UTC and was resolved at 2023-11-03 02:51 UTC, lasting 1 min; no cause or specific fix was stated.

CCourierMessage send delays

DDremioLogin to EU site unavailable

Dremio reported a major outage blocking password and oauth login to https://app.eu.dremio.cloud due to Auth0 problems; API access using PAT tokens was unaffected. The incident started at 2023-10-30 20:50 UTC and was resolved at 2023-10-30 21:14 UTC, lasting 23 min, with normal operation restored.

CCriblCribl.Cloud login unavailable

CockroachDBSingle sign-on to console is failing

CockroachDB reported a major Cloud Console authentication outage from 2023-10-30 20:25 UTC to 2023-10-30 20:36 UTC, lasting 11 min. An upstream provider outage caused all authentication methods except username and password to fail; existing sessions continued until timeout. The provider incident was resolved and SSO authentication restored.

PPipedreamPackage installation through npm yielding "Failed to fetch" errors

Pipedream reported a major incident affecting npm package installation: 500 errors could cause "Failed to fetch" errors when testing or deploying code steps using third-party npm packages. It started at 2023-10-30 20:19 UTC and was resolved at 2023-10-30 20:44 UTC, lasting 25 min; no cause or fix was stated.

StatuspageIntermittent errors while accessing public Statuspages

Statuspage recorded major intermittent public-page errors starting 2023-10-28 07:36 UTC, resolved at 2023-10-28 08:01 UTC (24 min). The postmortem reports all functionality affected in all regions from 06:00 UTC to 07:45 UTC on October 28, 2023, due to a database performance issue during scheduled database maintenance.

PostmanWorkspaces are not loading

Postman reported a major outage affecting workspace loading, listing Postman Platform on Browser and Postman Platform on Desktop as affected components. It started at 2023-10-26 12:16 UTC and was resolved at 2023-10-26 12:58 UTC, lasting 42 min. Postman reported that mitigation restored workspace loading and reduced error rates; no specific cause was stated.

KeeperGovCloud Vault Login

Keeper reported a critical GovCloud vault login incident affecting Keeper Admin Console (GovCloud) and Keeper Web Vault (GovCloud). It started at 2023-10-24 16:28 UTC and was resolved at 2023-10-24 16:33 UTC, lasting 5 min; no cause or specific fix was provided.

PPandaDocApplication is not available (502 Gateway, etc.)

PandaDoc reported a major application outage (502 Gateway, etc.) affecting EU and US & Global services, including APIs, document workflows and integrations. It started at 2023-10-23 00:42 UTC and was resolved at 2023-10-23 01:48 UTC, lasting 1 h 5 min. PandaDoc confirmed application access was restored but provided no specific cause or fix details.

Gustothat there is an issue that is currently causing longer than expected wait times for phone support. Our chat support is operating normally. We apologize

Gusto reported longer-than-expected phone support wait times, rated critical, while chat support operated normally. The incident started at 2023-10-19 14:05 UTC and was resolved at 2023-10-19 17:16 UTC, lasting 3 h 10 min.

Ccrates.ioFailing requests and timeouts

PyPI (Python Infrastructure)Database Storage Latencies

SSigma ComputingUsers unable to enter edits to Workbooks

Sigma Computing reported a major incident in which users who had recently edited a Workbook received errors when making further edits. The recorded start is 2023-10-06 20:27 UTC, but resolution is listed earlier, at 2023-08-09 22:00 UTC; duration is unknown. Engineering completed a postmortem and linked an RCA on Sigma Community.

AAblyably.com website, blog and control API down

Ably reported a major outage of ably.com, its blog and control API, with the website returning 500 errors. The incident started and was marked resolved at 2023-10-06 15:31 UTC (recorded duration: 0 min), although updates describe deploying a change at 03:34 PM UTC and services returning at 03:36 PM UTC.