What vendors changed

LLogRocketStreaming Data Export paused

LogRocket’s Streaming Data Export stopped exporting data in a critical incident starting 2023-09-22 16:35 UTC, lasting 3 h 24 min until resolution at 2023-09-22 19:59 UTC. LogRocket said exports would catch up on delayed data once reenabled and confirmed several successful exports before resolving the incident.

Jupyter (Anaconda)anaconda.cloud - Some channels degraded

Several anaconda.cloud channels returned 500 errors in a major-impact incident starting 2023-09-20 20:41 UTC and lasting 5 h 30 min, resolved at 2023-09-21 02:11 UTC. CLI calls were working while UI issues remained under investigation; Anaconda rolled out a fix and reported the platform fully operational after mitigation.

KeeperVault login issues in US

Keeper reported critical vault login issues affecting Keeper Web Vault (US) and Keeper Admin Console (US), from 2023-09-19 17:37 UTC to 2023-09-19 17:43 UTC (6 min). Its postmortem described intermittent API errors and identified bugs in the desktop app and backend API; DevOps resolved the issue.

LLightstep (ServiceNow Cloud Observability)app.lightstep.com was intermittently unavailable

DagsterServerless runs failing to launchEvent

Some Dagster Serverless runs failed to launch in a major outage starting 2023-09-18 18:07 UTC and resolved at 2023-09-19 04:03 UTC, lasting 9 h 55 min. Dagster reported AWS networking issues in two availability zones caused about 30% of new runs to fail container start; it deployed a fallback to stable zones, and the underlying issue appeared resolved.

DagsterDatabase maintenance failover

Dagster reported a critical incident starting 2023-09-18 12:46 UTC, with a computed duration of 0 min. The vendor described a 90-second disruption during cloud-provider database maintenance as a primary database failed over to a secondary; Dagster operations during that window would have failed. The incident was resolved, with all systems back to normal.

SSigma ComputingUnable to login and load workbooks for Sigma AWS US

Sigma Computing recorded a critical AWS US outage starting 2023-09-14 18:12 UTC that prevented users from logging in and loading workbooks. The incident lasted 2 h 7 min and was resolved at 2023-09-14 20:19 UTC; engineering implemented mitigation measures, but the root cause was not disclosed.

SSigma ComputingWorkbooks not loading for Sigma users on AWS Snowflake

Sigma Computing reported a major outage causing workbooks not to load and queries to hang for AWS US and AWS EU users using Snowflake CDW. The final update corrected the start to approximately 2023-09-14 12:49 UTC; mitigation resolved it at 2023-09-14 17:24 UTC (computed duration: 2 h 43 min).

StatuspageAtlassian Account login issues

Statuspage reported a major Atlassian Account login and signup incident affecting some Cloud customers and web portals, starting 2023-09-13 14:16 UTC and resolved 2023-09-13 19:37 UTC (5 h 21 min). A postmortem attributed sign-in and signup API failures to misconfigured rate limits in an internal service; mitigation restored access.

OpsgenieAtlassian Account login issues

Opsgenie reported a major Atlassian Account login and signup outage affecting some Cloud customers, including EU and US authentication components. The incident started at 2023-09-13 14:08 UTC and was resolved at 2023-09-13 19:32 UTC (5 h 23 min). The postmortem attributed sign-in and signup API failures to misconfigured internal rate limits; mitigation restored service.

MapboxMissing or Incorrect POI Brand Name in Search

Mapbox reported a major Search incident affecting select POI brands in the US and Germany: searches returned no results or POI names consisting only of numbers. It started at 2023-09-12 20:14 UTC and was resolved at 2023-09-12 21:23 UTC, lasting 1 h 9 min; Mapbox identified a fix and began a rollback.

CockroachDBConnectivity Issues

CockroachDB reported major connectivity issues isolated to its `Europe-West-1` Serverless cluster in GCP, starting 2023-09-08 15:47 UTC. A fix was implemented by 09-08 16:09 UTC, and the incident was resolved at 2023-09-08 17:01 UTC; the reported duration was 1 h 13 min.

DDremioIntermittent Query Failures

Dremio reported major intermittent query failures starting 2023-09-07 15:50 UTC, believed limited to AWS projects created since Aug 17th, 2023, following a deployment. Fixes restored normal operation in US and European production environments; the incident was resolved at 2023-09-08 17:40 UTC after 25 h 50 min.

SSigma ComputingWorkbooks cannot be opened for all Azure-hosted orgs

Sigma Computing reported a critical outage preventing all Azure-hosted organizations from opening workbooks, starting approximately 19:50 UTC Sep 6, 2023 and resolved at 2023-09-07 21:44 UTC (computed duration: 24 h 47 min). Azure Key Vault inaccessibility disrupted encryption/decryption; Key Vault recovery restored workbook and data warehouse connection access. Sigma stated customer metadata remained secure.

SStitchWeb application instability

Stitch reported a major availability incident affecting app.stitchdata.com from 2023-09-06 20:37 UTC to 2023-09-06 21:07 UTC, lasting 30 min. The team implemented a workaround, then resolved the root cause and confirmed stable availability; extraction and loading processes were unaffected, and no data was lost.

SSigma ComputingMaterialization failures impacting GCP cloud-hosted orgs

Sigma reported a major incident causing materialization and CSV upload failures for GCP-hosted customers, including GCP (US). Recorded from 2023-09-06 20:24 UTC to 2023-09-06 21:16 UTC, it lasted 52 min. Mitigation resolved the issue; affected materializations were expected to recover on their next scheduled run or when manually retriggered.

DagsterSome asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error

Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved.

CCloseElevated Error Rate

Close reported a major incident affecting API and Application UI, with elevated error rates and possible request timeouts. It started at 2023-09-05 16:08 UTC and was resolved at 2023-09-05 17:04 UTC; the recorded duration was 55 min. A fix was deployed by 16:17 UTC, restoring normal application operation.

TTabnineTabnine App website is not working properly

Tabnine reported a major outage affecting Management - Tabnine App and Tabnine App, with the management site partially available. The incident started at 2023-09-05 12:50 UTC and was resolved at 2023-09-05 15:55 UTC; the recorded duration was 3 h 4 min. Tabnine stated the problem was identified and fixed but provided no cause.

FFlagsmithCore API is not responding

MattermostIncident with Community

Mattermost Community experienced degraded availability in a major incident starting 2023-09-05 04:12 UTC and lasting 3 h 33 min, until resolution at 2023-09-05 07:45 UTC. Mattermost applied a mitigation at 09-05 07:29 UTC; no specific cause was stated.

AAblyTimeouts and High Latencies Across All Regions

Ably reported major timeouts and high latencies across all regions starting 2023-09-01 15:18 UTC. Computed duration was 0 min, although updates report disruption until 15:31 UTC. Initial reporting cited transient load causing resource starvation; the postmortem attributed the incident to a DDoS affecting a significant fraction of connections to primary and fallback endpoints.

HHexKernel Instability

Hex reported major kernel instability affecting Kernels starting 2023-08-30 17:34 UTC. The incident lasted 2 h 44 min and was resolved at 2023-08-30 20:18 UTC; no cause or specific fix was stated.

KeeperGovCloud Admin Console login with ARAM permissions

Keeper reported a major GovCloud Admin Console login outage affecting admins with Advanced Reporting & Alerts module (ARAM) permissions, from 2023-08-30 14:03 UTC to 2023-08-30 14:50 UTC (46 min). A subnet VPC routing change caused connectivity issues between backend services and a Redshift cluster; login was restored.

MezmoWeb UI is unresponsive and ingestion of log lines halted

Mezmo reported a major outage from 2023-08-29 21:01 UTC to 2023-08-30 00:17 UTC, lasting 3 h 16 min. Its Kong Gateway stopped functioning, preventing Web UI access and log ingestion through the agent or API; syslog ingestion was unaffected. The incident was resolved, but no fix was specified.

HHexKernel instability 08/29/23

Hex reported a major outage affecting Data Connections and Kernels, starting 2023-08-29 19:30 UTC and resolved at 2023-08-29 22:21 UTC, lasting 2 h 51 min. Hex said it identified the issue and worked with its hosting provider to resolve it, but did not state a specific cause or fix.

MattermostIncident with workspace logins

Mattermost reported a major incident affecting logins to Cloud Workspaces and Workspaces, starting at 2023-08-21 06:07 UTC and resolved at 2023-08-21 07:27 UTC after 1 h 20 min. A mitigation was applied at 06:54 UTC; no specific cause was stated.

LLogRocketIncreased App Latency

LogRocket reported a major Dashboard incident causing slow or unresponsive application load times, starting at 2023-08-16 20:08 UTC and resolved at 2023-08-16 22:35 UTC, lasting 2 h 27 min. The vendor identified the source of instability and worked through recovery but did not specify the cause or fix.

SSigma ComputingIncorrect timestamps returned from warehouse for some customers

Sigma Computing reported a major incident affecting some customers: warehouse timestamps were off by their UTC offset, and date_trunc() could return incorrect values due to code refactoring. The recorded start is 2023-08-14 17:34 UTC, but resolution is dated earlier, 2023-08-07 17:32 UTC; duration is unknown.

SSigma ComputingExports with Color Conditional Formatting of "None" Failing

Sigma Computing reported a major (Severity 0) incident affecting Excel and CSV exports from some workbooks with Color Conditional Formatting of "None". It started at 2023-08-14 14:52 UTC and was marked resolved at 2023-08-14 17:07 UTC, lasting 2 h 15 min. Sigma later published a postmortem link; no cause or fix is detailed in the evidence.

DDailyIssue with sessions in ap-northeast-2

GGrainRecorder failing to join some Meet meetings

Grain Recorder failed to join some Meet meetings in a critical incident starting 2023-08-08 06:51 UTC and resolved 2024-01-19 17:41 UTC, lasting 164 days 10 h. Grain suspected a new Google anti-spam error state and deployed an update to Google Meet authenticated recorders to work around Google's limitations.

CCriblNo Assignable organizations in us-west2

DDopplerUnable to fetch secrets from the Doppler dashboard and API

Doppler reported a major outage preventing customers from fetching secrets through the dashboard and API, starting 2023-08-04 22:28 UTC and resolved at 2023-08-04 22:50 UTC (22 min). The vendor attributed the incident to a faulty Kubernetes NetworkPolicy change and said it would evaluate deployment procedures to catch such changes.

DagsterCannot launch runs in Serverless

Dagster reported a major API incident preventing Serverless runs from launching, starting 2023-08-03 20:29 UTC and resolved at 2023-08-03 21:23 UTC (53 min). Runs were launching successfully as of 20:50 UTC; Dagster attributed the issue to AWS EC2 API errors and latency affecting AWS Fargate, used by Dagster Cloud Serverless.

CockroachDBServerless intermittent connection failures

CockroachDB Serverless clusters experienced intermittent connection failures in AWS us-east-1, with major impact, starting 2023-08-03 01:12 UTC. The incident lasted 2 h 40 min and was resolved at 2023-08-03 03:53 UTC after engineers deployed a fix.

CockroachDBServerless Cluster Resource Limit Throttling

CockroachDB reported a major Serverless degradation in AWS eu-west-1 and AWS ap-south-1: some clusters remained throttled after resource limits were increased. It started at 2023-08-02 16:05 UTC and was resolved at 2023-08-02 19:44 UTC, lasting 3 h 39 min; previously throttled clusters now reflect resource-limit changes.

FFireboltPotential access issue to the UI.

Firebolt reported a critical incident potentially affecting UI access and, in a later update, programmatic connections. It started at 2023-08-02 12:25 UTC and was resolved at 2023-08-02 13:00 UTC, lasting 35 min; Firebolt confirmed all services were operating as expected.

PyPI (Python Infrastructure)Some PyPI web requests failing.

CCloseElevated Error Rate

Close reported major degradation and elevated errors affecting its application and API between 16:47 UTC and 17:18 UTC on July 26, 2023. The recorded start, 2023-07-28 15:45 UTC, follows the listed resolution, 2023-07-26 16:30 UTC; computed duration is unknown. No cause or fix is shown.

Metabaseand MMetabase CloudUnable to create new Metabase cloud instances

Customers were unable to create new Metabase cloud trials or cloud instances in a critical incident affecting Metabase Store. It started at 2023-07-28 13:10 UTC and was resolved at 2023-07-29 10:21 UTC, lasting 21 h 10 min; no cause or fix was stated.

MMetabase Cloudand MetabaseUnable to create new database connections

Metabase Cloud customers could not add new databases or edit database connections during a major incident affecting Metabase Cloud Platform. It started at 2023-07-28 13:01 UTC and was resolved at 2023-07-29 09:06 UTC, lasting 20 h 4 min; no cause or specific fix was stated.

SSigma ComputingScheduled exports using image (PNG) files erroring

Sigma Computing reported a major (Severity 0) incident affecting all scheduled export types, with errors later described as intermittent across AWS, Azure and GCP. It started at 2023-07-27 13:29 UTC and was resolved at 2023-07-27 14:13 UTC, lasting 44 min. Sigma implemented a fix and said customers could resend impacted exports.

RootlySlack Outage (3rd Party) Impacting Rootly Slack App

A major Slack outage affected Rootly’s Slack app, including channel creation and notifications, from 2023-07-27 09:00 UTC to 2023-07-27 10:05 UTC (1 h 4 min). Rootly.com’s Web App remained operational for incident creation; Rootly’s services were fully restored after Slack restored functionality.

CCourierInbox messages degradation for versions lower than v2

CockroachDBServerless clients unable to access GCP Multi-Region Serverless Public Preview hosts

CockroachDB reported a major outage affecting connections to Serverless clusters on the GCP Multi-Region Serverless Public Preview host; other serverless clusters remained fully functional. The incident started at 2023-07-25 21:55 UTC and was resolved at 2023-07-25 23:07 UTC, lasting 1 h 12 min; no cause or specific fix was stated.

KeeperLogin Errors - US Production

Keeper reported critical US Production login errors affecting Keeper Admin Console (US) and Keeper Web Vault (US), from 2023-07-25 20:53 UTC to 2023-07-25 20:57 UTC (4 min). Errors followed a scheduled backend API software update and resolved when it completed; the underlying cause remained under investigation.

MattermostIncident with Community

Mattermost reported degraded availability for Community with major impact, starting at 2023-07-25 10:37 UTC and resolved at 2023-07-25 11:15 UTC; the computed duration was 37 min. A mitigation was applied at 11:10 UTC, but no specific cause was stated.