What vendors changed

SSigma ComputingWorkbooks cannot be opened for all Azure-hosted orgs

Sigma Computing reported a critical outage preventing all Azure-hosted organizations from opening workbooks, starting approximately 19:50 UTC Sep 6, 2023 and resolved at 2023-09-07 21:44 UTC (computed duration: 24 h 47 min). Azure Key Vault inaccessibility disrupted encryption/decryption; Key Vault recovery restored workbook and data warehouse connection access. Sigma stated customer metadata remained secure.

SStitchWeb application instability

Stitch reported a major availability incident affecting app.stitchdata.com from 2023-09-06 20:37 UTC to 2023-09-06 21:07 UTC, lasting 30 min. The team implemented a workaround, then resolved the root cause and confirmed stable availability; extraction and loading processes were unaffected, and no data was lost.

SSigma ComputingMaterialization failures impacting GCP cloud-hosted orgs

Sigma reported a major incident causing materialization and CSV upload failures for GCP-hosted customers, including GCP (US). Recorded from 2023-09-06 20:24 UTC to 2023-09-06 21:16 UTC, it lasted 52 min. Mitigation resolved the issue; affected materializations were expected to recover on their next scheduled run or when manually retriggered.

DagsterSome asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error

Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved.

CCloseElevated Error Rate

Close reported a major incident affecting API and Application UI, with elevated error rates and possible request timeouts. It started at 2023-09-05 16:08 UTC and was resolved at 2023-09-05 17:04 UTC; the recorded duration was 55 min. A fix was deployed by 16:17 UTC, restoring normal application operation.

TTabnineTabnine App website is not working properly

Tabnine reported a major outage affecting Management - Tabnine App and Tabnine App, with the management site partially available. The incident started at 2023-09-05 12:50 UTC and was resolved at 2023-09-05 15:55 UTC; the recorded duration was 3 h 4 min. Tabnine stated the problem was identified and fixed but provided no cause.

FFlagsmithCore API is not responding

MattermostIncident with Community

Mattermost Community experienced degraded availability in a major incident starting 2023-09-05 04:12 UTC and lasting 3 h 33 min, until resolution at 2023-09-05 07:45 UTC. Mattermost applied a mitigation at 09-05 07:29 UTC; no specific cause was stated.

AAblyTimeouts and High Latencies Across All Regions

Ably reported major timeouts and high latencies across all regions starting 2023-09-01 15:18 UTC. Computed duration was 0 min, although updates report disruption until 15:31 UTC. Initial reporting cited transient load causing resource starvation; the postmortem attributed the incident to a DDoS affecting a significant fraction of connections to primary and fallback endpoints.

HHexKernel Instability

Hex reported major kernel instability affecting Kernels starting 2023-08-30 17:34 UTC. The incident lasted 2 h 44 min and was resolved at 2023-08-30 20:18 UTC; no cause or specific fix was stated.

KeeperGovCloud Admin Console login with ARAM permissions

Keeper reported a major GovCloud Admin Console login outage affecting admins with Advanced Reporting & Alerts module (ARAM) permissions, from 2023-08-30 14:03 UTC to 2023-08-30 14:50 UTC (46 min). A subnet VPC routing change caused connectivity issues between backend services and a Redshift cluster; login was restored.

MezmoWeb UI is unresponsive and ingestion of log lines halted

Mezmo reported a major outage from 2023-08-29 21:01 UTC to 2023-08-30 00:17 UTC, lasting 3 h 16 min. Its Kong Gateway stopped functioning, preventing Web UI access and log ingestion through the agent or API; syslog ingestion was unaffected. The incident was resolved, but no fix was specified.

HHexKernel instability 08/29/23

Hex reported a major outage affecting Data Connections and Kernels, starting 2023-08-29 19:30 UTC and resolved at 2023-08-29 22:21 UTC, lasting 2 h 51 min. Hex said it identified the issue and worked with its hosting provider to resolve it, but did not state a specific cause or fix.

MattermostIncident with workspace logins

Mattermost reported a major incident affecting logins to Cloud Workspaces and Workspaces, starting at 2023-08-21 06:07 UTC and resolved at 2023-08-21 07:27 UTC after 1 h 20 min. A mitigation was applied at 06:54 UTC; no specific cause was stated.

LLogRocketIncreased App Latency

LogRocket reported a major Dashboard incident causing slow or unresponsive application load times, starting at 2023-08-16 20:08 UTC and resolved at 2023-08-16 22:35 UTC, lasting 2 h 27 min. The vendor identified the source of instability and worked through recovery but did not specify the cause or fix.

SSigma ComputingIncorrect timestamps returned from warehouse for some customers

Sigma Computing reported a major incident affecting some customers: warehouse timestamps were off by their UTC offset, and date_trunc() could return incorrect values due to code refactoring. The recorded start is 2023-08-14 17:34 UTC, but resolution is dated earlier, 2023-08-07 17:32 UTC; duration is unknown.

SSigma ComputingExports with Color Conditional Formatting of "None" Failing

Sigma Computing reported a major (Severity 0) incident affecting Excel and CSV exports from some workbooks with Color Conditional Formatting of "None". It started at 2023-08-14 14:52 UTC and was marked resolved at 2023-08-14 17:07 UTC, lasting 2 h 15 min. Sigma later published a postmortem link; no cause or fix is detailed in the evidence.

DDailyIssue with sessions in ap-northeast-2

GGrainRecorder failing to join some Meet meetings

Grain Recorder failed to join some Meet meetings in a critical incident starting 2023-08-08 06:51 UTC and resolved 2024-01-19 17:41 UTC, lasting 164 days 10 h. Grain suspected a new Google anti-spam error state and deployed an update to Google Meet authenticated recorders to work around Google's limitations.

CCriblNo Assignable organizations in us-west2

DDopplerUnable to fetch secrets from the Doppler dashboard and API

Doppler reported a major outage preventing customers from fetching secrets through the dashboard and API, starting 2023-08-04 22:28 UTC and resolved at 2023-08-04 22:50 UTC (22 min). The vendor attributed the incident to a faulty Kubernetes NetworkPolicy change and said it would evaluate deployment procedures to catch such changes.

DagsterCannot launch runs in Serverless

Dagster reported a major API incident preventing Serverless runs from launching, starting 2023-08-03 20:29 UTC and resolved at 2023-08-03 21:23 UTC (53 min). Runs were launching successfully as of 20:50 UTC; Dagster attributed the issue to AWS EC2 API errors and latency affecting AWS Fargate, used by Dagster Cloud Serverless.

CockroachDBServerless intermittent connection failures

CockroachDB Serverless clusters experienced intermittent connection failures in AWS us-east-1, with major impact, starting 2023-08-03 01:12 UTC. The incident lasted 2 h 40 min and was resolved at 2023-08-03 03:53 UTC after engineers deployed a fix.

CockroachDBServerless Cluster Resource Limit Throttling

CockroachDB reported a major Serverless degradation in AWS eu-west-1 and AWS ap-south-1: some clusters remained throttled after resource limits were increased. It started at 2023-08-02 16:05 UTC and was resolved at 2023-08-02 19:44 UTC, lasting 3 h 39 min; previously throttled clusters now reflect resource-limit changes.

FFireboltPotential access issue to the UI.

Firebolt reported a critical incident potentially affecting UI access and, in a later update, programmatic connections. It started at 2023-08-02 12:25 UTC and was resolved at 2023-08-02 13:00 UTC, lasting 35 min; Firebolt confirmed all services were operating as expected.

PyPI (Python Infrastructure)Some PyPI web requests failing.

CCloseElevated Error Rate

Close reported major degradation and elevated errors affecting its application and API between 16:47 UTC and 17:18 UTC on July 26, 2023. The recorded start, 2023-07-28 15:45 UTC, follows the listed resolution, 2023-07-26 16:30 UTC; computed duration is unknown. No cause or fix is shown.

Metabaseand MMetabase CloudUnable to create new Metabase cloud instances

Customers were unable to create new Metabase cloud trials or cloud instances in a critical incident affecting Metabase Store. It started at 2023-07-28 13:10 UTC and was resolved at 2023-07-29 10:21 UTC, lasting 21 h 10 min; no cause or fix was stated.

MMetabase Cloudand MetabaseUnable to create new database connections

Metabase Cloud customers could not add new databases or edit database connections during a major incident affecting Metabase Cloud Platform. It started at 2023-07-28 13:01 UTC and was resolved at 2023-07-29 09:06 UTC, lasting 20 h 4 min; no cause or specific fix was stated.

SSigma ComputingScheduled exports using image (PNG) files erroring

Sigma Computing reported a major (Severity 0) incident affecting all scheduled export types, with errors later described as intermittent across AWS, Azure and GCP. It started at 2023-07-27 13:29 UTC and was resolved at 2023-07-27 14:13 UTC, lasting 44 min. Sigma implemented a fix and said customers could resend impacted exports.

RootlySlack Outage (3rd Party) Impacting Rootly Slack App

A major Slack outage affected Rootly’s Slack app, including channel creation and notifications, from 2023-07-27 09:00 UTC to 2023-07-27 10:05 UTC (1 h 4 min). Rootly.com’s Web App remained operational for incident creation; Rootly’s services were fully restored after Slack restored functionality.

CCourierInbox messages degradation for versions lower than v2

CockroachDBServerless clients unable to access GCP Multi-Region Serverless Public Preview hosts

CockroachDB reported a major outage affecting connections to Serverless clusters on the GCP Multi-Region Serverless Public Preview host; other serverless clusters remained fully functional. The incident started at 2023-07-25 21:55 UTC and was resolved at 2023-07-25 23:07 UTC, lasting 1 h 12 min; no cause or specific fix was stated.

KeeperLogin Errors - US Production

Keeper reported critical US Production login errors affecting Keeper Admin Console (US) and Keeper Web Vault (US), from 2023-07-25 20:53 UTC to 2023-07-25 20:57 UTC (4 min). Errors followed a scheduled backend API software update and resolved when it completed; the underlying cause remained under investigation.

MattermostIncident with Community

Mattermost reported degraded availability for Community with major impact, starting at 2023-07-25 10:37 UTC and resolved at 2023-07-25 11:15 UTC; the computed duration was 37 min. A mitigation was applied at 11:10 UTC, but no specific cause was stated.

CCourierDelay in the messages sent out

LLogRocketLogRocket dashboard degraded performance

LogRocket reported a major degradation affecting Dashboard and Issues starting 2023-07-20 22:30 UTC, resolved at 2023-07-21 00:27 UTC after 1 h 56 min. Users could experience missing sourcemap uploads; data collection was unaffected. LogRocket identified the issue and implemented a fix, but stated no cause.

Ccrates.ioBroken download links

DDremioDatabase Connectivity

Dremio reported a major database connectivity incident affecting Dremio Cloud - US - Site Availability and Site Availability. It started and was resolved at 2023-07-18 22:42 UTC, with a recorded duration of 0 min; Dremio said it mitigated the connectivity issue and database connections recovered.

DDropbox SignSite Outage

Dropbox Sign reported a major site outage affecting signing, signature requests, document delivery, API callbacks, Dropbox Forms, fax services, and HubSpot and Salesforce integrations. It started at 2023-07-13 20:49 UTC and was resolved at 2023-07-13 20:53 UTC, with a computed duration of 3 min; no cause or fix was stated.

PostmanUpstream dependency issues for Bitbucket

Postman reported a major outage affecting Postman Integrations, with upstream dependency issues disrupting Integration APIs for Bitbucket. It started at 2023-07-13 04:51 UTC and was resolved at 2023-07-13 07:45 UTC, lasting 2 h 54 min.

MapboxDegraded Live Traffic Data.

Mapbox reported critical live traffic degradation starting 2023-07-13 00:55 UTC, lasting 2 h 17 min until resolution at 2023-07-13 03:12 UTC. Customers using Navigation APIs powered by Mapbox traffic could receive inaccurate ETAs, while traffic data customers could receive intermittent live updates in impacted regions. A fix was implemented at 07-13 01:53 UTC.

LLightstep (ServiceNow Cloud Observability)Lightstep experience major issues with web UI, query and ingest.

OpsgenieIntermittent errors during login for some customers

Opsgenie reported a major incident causing intermittent login errors for some customers, with EU and US Signup, Login & Authorization components affected. The incident started at 2023-07-06 09:03 UTC and was resolved at 2023-07-06 13:32 UTC, lasting 4 h 29 min; the vendor said it identified the root cause and mitigated the problem.

IimgixIssue with deploying new/updated Source configurations

IimgixElevated rendering errors

GustoMost users are experiencing errors within their Gusto account

Most Gusto users experienced account errors affecting Payroll, Benefits, and HR in a major outage starting 2023-06-30 17:52 UTC. Gusto implemented a fix by 2023-06-30 18:00 UTC and resolved the incident at 2023-06-30 19:00 UTC; it lasted 1 h 8 min.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site from 2023-06-26 16:10 UTC to 2023-06-26 16:25 UTC, lasting 15 min. A fix was implemented by 2023-06-26 16:20 UTC, and the incident was resolved.