What vendors changed

CCriblCribl.Cloud login unavailable

CockroachDBSingle sign-on to console is failing

CockroachDB reported a major Cloud Console authentication outage from 2023-10-30 20:25 UTC to 2023-10-30 20:36 UTC, lasting 11 min. An upstream provider outage caused all authentication methods except username and password to fail; existing sessions continued until timeout. The provider incident was resolved and SSO authentication restored.

PPipedreamPackage installation through npm yielding "Failed to fetch" errors

Pipedream reported a major incident affecting npm package installation: 500 errors could cause "Failed to fetch" errors when testing or deploying code steps using third-party npm packages. It started at 2023-10-30 20:19 UTC and was resolved at 2023-10-30 20:44 UTC, lasting 25 min; no cause or fix was stated.

StatuspageIntermittent errors while accessing public Statuspages

Statuspage recorded major intermittent public-page errors starting 2023-10-28 07:36 UTC, resolved at 2023-10-28 08:01 UTC (24 min). The postmortem reports all functionality affected in all regions from 06:00 UTC to 07:45 UTC on October 28, 2023, due to a database performance issue during scheduled database maintenance.

PostmanWorkspaces are not loading

Postman reported a major outage affecting workspace loading, listing Postman Platform on Browser and Postman Platform on Desktop as affected components. It started at 2023-10-26 12:16 UTC and was resolved at 2023-10-26 12:58 UTC, lasting 42 min. Postman reported that mitigation restored workspace loading and reduced error rates; no specific cause was stated.

KeeperGovCloud Vault Login

Keeper reported a critical GovCloud vault login incident affecting Keeper Admin Console (GovCloud) and Keeper Web Vault (GovCloud). It started at 2023-10-24 16:28 UTC and was resolved at 2023-10-24 16:33 UTC, lasting 5 min; no cause or specific fix was provided.

PPandaDocApplication is not available (502 Gateway, etc.)

PandaDoc reported a major application outage (502 Gateway, etc.) affecting EU and US & Global services, including APIs, document workflows and integrations. It started at 2023-10-23 00:42 UTC and was resolved at 2023-10-23 01:48 UTC, lasting 1 h 5 min. PandaDoc confirmed application access was restored but provided no specific cause or fix details.

Gustothat there is an issue that is currently causing longer than expected wait times for phone support. Our chat support is operating normally. We apologize

Gusto reported longer-than-expected phone support wait times, rated critical, while chat support operated normally. The incident started at 2023-10-19 14:05 UTC and was resolved at 2023-10-19 17:16 UTC, lasting 3 h 10 min.

Ccrates.ioFailing requests and timeouts

PyPI (Python Infrastructure)Database Storage Latencies

SSigma ComputingUsers unable to enter edits to Workbooks

Sigma Computing reported a major incident in which users who had recently edited a Workbook received errors when making further edits. The recorded start is 2023-10-06 20:27 UTC, but resolution is listed earlier, at 2023-08-09 22:00 UTC; duration is unknown. Engineering completed a postmortem and linked an RCA on Sigma Community.

AAblyably.com website, blog and control API down

Ably reported a major outage of ably.com, its blog and control API, with the website returning 500 errors. The incident started and was marked resolved at 2023-10-06 15:31 UTC (recorded duration: 0 min), although updates describe deploying a change at 03:34 PM UTC and services returning at 03:36 PM UTC.

HHexDegraded kernel execution

Hex reported degraded kernel execution affecting Kernels and Main site, with major impact, starting at 2023-10-04 18:14 UTC. A fix was implemented by 2023-10-04 18:31 UTC, and the incident was resolved at 2023-10-04 18:45 UTC after 31 min.

LLightdashDegraded performance and Snowflake Query Failures

Lightdash reported a major incident affecting Lightdash Cloud (EU) and Lightdash Cloud (US), with Snowflake query failures, dashboards not loading charts, app crashes and high backend latency. It started and was marked resolved at 2023-10-03 15:39 UTC (recorded duration: 0 min); Lightdash said it identified the root cause and took corrective measures but provided no specifics.

SSigma ComputingQuery failures for select organizations connected to Snowflake

Sigma Computing reported a critical incident affecting Snowflake queries for a subset of users on AWS and GCP (US), returning "Bad request; operation not supported." It lasted 5 h 7 min, from 2023-10-03 13:41 UTC to 2023-10-03 18:48 UTC; Snowflake rolled back a server change globally, restoring access to workbooks and data warehouse connections.

FFireboltEngine Startup Issue in US-EAST-1 RegionEvent

Firebolt reported a major engine startup outage in US-EAST-1 caused by an AWS issue, primarily affecting new instance launches; running engines and other regions were unaffected. It started at 2023-09-28 20:10 UTC and was resolved at 2023-09-29 06:37 UTC, lasting 10 h 26 min, with normal service restored.

AAblyably.com and support.ably.com down

Ably reported a major outage affecting ably.com, support.ably.com, customer dashboards, blog and CMS, citing Heroku issues; realtime services were unaffected. The incident started and was marked resolved at 2023-09-27 18:09 UTC, with a recorded duration of 0 min, although the update still described website issues.

PPandaDocAPI Outage

PandaDoc’s critical API outage started at 2023-09-26 18:29 UTC and lasted 58 min, resolving at 2023-09-26 19:27 UTC. A fix was implemented at 18:59 UTC; the postmortem said some users’ API keys were temporarily deactivated, requiring new ones, and stated PandaDoc was not breached or hacked.

LLogRocketStreaming Data Export paused

LogRocket’s Streaming Data Export stopped exporting data in a critical incident starting 2023-09-22 16:35 UTC, lasting 3 h 24 min until resolution at 2023-09-22 19:59 UTC. LogRocket said exports would catch up on delayed data once reenabled and confirmed several successful exports before resolving the incident.

Jupyter (Anaconda)anaconda.cloud - Some channels degraded

Several anaconda.cloud channels returned 500 errors in a major-impact incident starting 2023-09-20 20:41 UTC and lasting 5 h 30 min, resolved at 2023-09-21 02:11 UTC. CLI calls were working while UI issues remained under investigation; Anaconda rolled out a fix and reported the platform fully operational after mitigation.

KeeperVault login issues in US

Keeper reported critical vault login issues affecting Keeper Web Vault (US) and Keeper Admin Console (US), from 2023-09-19 17:37 UTC to 2023-09-19 17:43 UTC (6 min). Its postmortem described intermittent API errors and identified bugs in the desktop app and backend API; DevOps resolved the issue.

LLightstep (ServiceNow Cloud Observability)app.lightstep.com was intermittently unavailable

DagsterServerless runs failing to launchEvent

Some Dagster Serverless runs failed to launch in a major outage starting 2023-09-18 18:07 UTC and resolved at 2023-09-19 04:03 UTC, lasting 9 h 55 min. Dagster reported AWS networking issues in two availability zones caused about 30% of new runs to fail container start; it deployed a fallback to stable zones, and the underlying issue appeared resolved.

DagsterDatabase maintenance failover

Dagster reported a critical incident starting 2023-09-18 12:46 UTC, with a computed duration of 0 min. The vendor described a 90-second disruption during cloud-provider database maintenance as a primary database failed over to a secondary; Dagster operations during that window would have failed. The incident was resolved, with all systems back to normal.

SSigma ComputingUnable to login and load workbooks for Sigma AWS US

Sigma Computing recorded a critical AWS US outage starting 2023-09-14 18:12 UTC that prevented users from logging in and loading workbooks. The incident lasted 2 h 7 min and was resolved at 2023-09-14 20:19 UTC; engineering implemented mitigation measures, but the root cause was not disclosed.

SSigma ComputingWorkbooks not loading for Sigma users on AWS Snowflake

Sigma Computing reported a major outage causing workbooks not to load and queries to hang for AWS US and AWS EU users using Snowflake CDW. The final update corrected the start to approximately 2023-09-14 12:49 UTC; mitigation resolved it at 2023-09-14 17:24 UTC (computed duration: 2 h 43 min).

StatuspageAtlassian Account login issues

Statuspage reported a major Atlassian Account login and signup incident affecting some Cloud customers and web portals, starting 2023-09-13 14:16 UTC and resolved 2023-09-13 19:37 UTC (5 h 21 min). A postmortem attributed sign-in and signup API failures to misconfigured rate limits in an internal service; mitigation restored access.

OpsgenieAtlassian Account login issues

Opsgenie reported a major Atlassian Account login and signup outage affecting some Cloud customers, including EU and US authentication components. The incident started at 2023-09-13 14:08 UTC and was resolved at 2023-09-13 19:32 UTC (5 h 23 min). The postmortem attributed sign-in and signup API failures to misconfigured internal rate limits; mitigation restored service.

MapboxMissing or Incorrect POI Brand Name in Search

Mapbox reported a major Search incident affecting select POI brands in the US and Germany: searches returned no results or POI names consisting only of numbers. It started at 2023-09-12 20:14 UTC and was resolved at 2023-09-12 21:23 UTC, lasting 1 h 9 min; Mapbox identified a fix and began a rollback.

CockroachDBConnectivity Issues

CockroachDB reported major connectivity issues isolated to its `Europe-West-1` Serverless cluster in GCP, starting 2023-09-08 15:47 UTC. A fix was implemented by 09-08 16:09 UTC, and the incident was resolved at 2023-09-08 17:01 UTC; the reported duration was 1 h 13 min.

DDremioIntermittent Query Failures

Dremio reported major intermittent query failures starting 2023-09-07 15:50 UTC, believed limited to AWS projects created since Aug 17th, 2023, following a deployment. Fixes restored normal operation in US and European production environments; the incident was resolved at 2023-09-08 17:40 UTC after 25 h 50 min.

SSigma ComputingWorkbooks cannot be opened for all Azure-hosted orgs

Sigma Computing reported a critical outage preventing all Azure-hosted organizations from opening workbooks, starting approximately 19:50 UTC Sep 6, 2023 and resolved at 2023-09-07 21:44 UTC (computed duration: 24 h 47 min). Azure Key Vault inaccessibility disrupted encryption/decryption; Key Vault recovery restored workbook and data warehouse connection access. Sigma stated customer metadata remained secure.

SStitchWeb application instability

Stitch reported a major availability incident affecting app.stitchdata.com from 2023-09-06 20:37 UTC to 2023-09-06 21:07 UTC, lasting 30 min. The team implemented a workaround, then resolved the root cause and confirmed stable availability; extraction and loading processes were unaffected, and no data was lost.

SSigma ComputingMaterialization failures impacting GCP cloud-hosted orgs

Sigma reported a major incident causing materialization and CSV upload failures for GCP-hosted customers, including GCP (US). Recorded from 2023-09-06 20:24 UTC to 2023-09-06 21:16 UTC, it lasted 52 min. Mitigation resolved the issue; affected materializations were expected to recover on their next scheduled run or when manually retriggered.

DagsterSome asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error

Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved.

CCloseElevated Error Rate

Close reported a major incident affecting API and Application UI, with elevated error rates and possible request timeouts. It started at 2023-09-05 16:08 UTC and was resolved at 2023-09-05 17:04 UTC; the recorded duration was 55 min. A fix was deployed by 16:17 UTC, restoring normal application operation.

TTabnineTabnine App website is not working properly

Tabnine reported a major outage affecting Management - Tabnine App and Tabnine App, with the management site partially available. The incident started at 2023-09-05 12:50 UTC and was resolved at 2023-09-05 15:55 UTC; the recorded duration was 3 h 4 min. Tabnine stated the problem was identified and fixed but provided no cause.

FFlagsmithCore API is not responding

MattermostIncident with Community

Mattermost Community experienced degraded availability in a major incident starting 2023-09-05 04:12 UTC and lasting 3 h 33 min, until resolution at 2023-09-05 07:45 UTC. Mattermost applied a mitigation at 09-05 07:29 UTC; no specific cause was stated.

AAblyTimeouts and High Latencies Across All Regions

Ably reported major timeouts and high latencies across all regions starting 2023-09-01 15:18 UTC. Computed duration was 0 min, although updates report disruption until 15:31 UTC. Initial reporting cited transient load causing resource starvation; the postmortem attributed the incident to a DDoS affecting a significant fraction of connections to primary and fallback endpoints.

HHexKernel Instability

Hex reported major kernel instability affecting Kernels starting 2023-08-30 17:34 UTC. The incident lasted 2 h 44 min and was resolved at 2023-08-30 20:18 UTC; no cause or specific fix was stated.

KeeperGovCloud Admin Console login with ARAM permissions

Keeper reported a major GovCloud Admin Console login outage affecting admins with Advanced Reporting & Alerts module (ARAM) permissions, from 2023-08-30 14:03 UTC to 2023-08-30 14:50 UTC (46 min). A subnet VPC routing change caused connectivity issues between backend services and a Redshift cluster; login was restored.

MezmoWeb UI is unresponsive and ingestion of log lines halted

Mezmo reported a major outage from 2023-08-29 21:01 UTC to 2023-08-30 00:17 UTC, lasting 3 h 16 min. Its Kong Gateway stopped functioning, preventing Web UI access and log ingestion through the agent or API; syslog ingestion was unaffected. The incident was resolved, but no fix was specified.

HHexKernel instability 08/29/23

Hex reported a major outage affecting Data Connections and Kernels, starting 2023-08-29 19:30 UTC and resolved at 2023-08-29 22:21 UTC, lasting 2 h 51 min. Hex said it identified the issue and worked with its hosting provider to resolve it, but did not state a specific cause or fix.

MattermostIncident with workspace logins

Mattermost reported a major incident affecting logins to Cloud Workspaces and Workspaces, starting at 2023-08-21 06:07 UTC and resolved at 2023-08-21 07:27 UTC after 1 h 20 min. A mitigation was applied at 06:54 UTC; no specific cause was stated.

LLogRocketIncreased App Latency

LogRocket reported a major Dashboard incident causing slow or unresponsive application load times, starting at 2023-08-16 20:08 UTC and resolved at 2023-08-16 22:35 UTC, lasting 2 h 27 min. The vendor identified the source of instability and worked through recovery but did not specify the cause or fix.

SSigma ComputingIncorrect timestamps returned from warehouse for some customers

Sigma Computing reported a major incident affecting some customers: warehouse timestamps were off by their UTC offset, and date_trunc() could return incorrect values due to code refactoring. The recorded start is 2023-08-14 17:34 UTC, but resolution is dated earlier, 2023-08-07 17:32 UTC; duration is unknown.

SSigma ComputingExports with Color Conditional Formatting of "None" Failing

Sigma Computing reported a major (Severity 0) incident affecting Excel and CSV exports from some workbooks with Color Conditional Formatting of "None". It started at 2023-08-14 14:52 UTC and was marked resolved at 2023-08-14 17:07 UTC, lasting 2 h 15 min. Sigma later published a postmortem link; no cause or fix is detailed in the evidence.

DDailyIssue with sessions in ap-northeast-2

GGrainRecorder failing to join some Meet meetings

Grain Recorder failed to join some Meet meetings in a critical incident starting 2023-08-08 06:51 UTC and resolved 2024-01-19 17:41 UTC, lasting 164 days 10 h. Grain suspected a new Google anti-spam error state and deployed an update to Google Meet authenticated recorders to work around Google's limitations.