What vendors changed

SSigma ComputingIncorrect timestamps returned from warehouse for some customers

Sigma Computing reported a major incident affecting some customers: warehouse timestamps were off by their UTC offset, and date_trunc() could return incorrect values due to code refactoring. The recorded start is 2023-08-14 17:34 UTC, but resolution is dated earlier, 2023-08-07 17:32 UTC; duration is unknown.

SSigma ComputingExports with Color Conditional Formatting of "None" Failing

Sigma Computing reported a major (Severity 0) incident affecting Excel and CSV exports from some workbooks with Color Conditional Formatting of "None". It started at 2023-08-14 14:52 UTC and was marked resolved at 2023-08-14 17:07 UTC, lasting 2 h 15 min. Sigma later published a postmortem link; no cause or fix is detailed in the evidence.

DDailyIssue with sessions in ap-northeast-2

GGrainRecorder failing to join some Meet meetings

Grain Recorder failed to join some Meet meetings in a critical incident starting 2023-08-08 06:51 UTC and resolved 2024-01-19 17:41 UTC, lasting 164 days 10 h. Grain suspected a new Google anti-spam error state and deployed an update to Google Meet authenticated recorders to work around Google's limitations.

CCriblNo Assignable organizations in us-west2

DDopplerUnable to fetch secrets from the Doppler dashboard and API

Doppler reported a major outage preventing customers from fetching secrets through the dashboard and API, starting 2023-08-04 22:28 UTC and resolved at 2023-08-04 22:50 UTC (22 min). The vendor attributed the incident to a faulty Kubernetes NetworkPolicy change and said it would evaluate deployment procedures to catch such changes.

DagsterCannot launch runs in Serverless

Dagster reported a major API incident preventing Serverless runs from launching, starting 2023-08-03 20:29 UTC and resolved at 2023-08-03 21:23 UTC (53 min). Runs were launching successfully as of 20:50 UTC; Dagster attributed the issue to AWS EC2 API errors and latency affecting AWS Fargate, used by Dagster Cloud Serverless.

CockroachDBServerless intermittent connection failures

CockroachDB Serverless clusters experienced intermittent connection failures in AWS us-east-1, with major impact, starting 2023-08-03 01:12 UTC. The incident lasted 2 h 40 min and was resolved at 2023-08-03 03:53 UTC after engineers deployed a fix.

CockroachDBServerless Cluster Resource Limit Throttling

CockroachDB reported a major Serverless degradation in AWS eu-west-1 and AWS ap-south-1: some clusters remained throttled after resource limits were increased. It started at 2023-08-02 16:05 UTC and was resolved at 2023-08-02 19:44 UTC, lasting 3 h 39 min; previously throttled clusters now reflect resource-limit changes.

FFireboltPotential access issue to the UI.

Firebolt reported a critical incident potentially affecting UI access and, in a later update, programmatic connections. It started at 2023-08-02 12:25 UTC and was resolved at 2023-08-02 13:00 UTC, lasting 35 min; Firebolt confirmed all services were operating as expected.

PyPI (Python Infrastructure)Some PyPI web requests failing.

CCloseElevated Error Rate

Close reported major degradation and elevated errors affecting its application and API between 16:47 UTC and 17:18 UTC on July 26, 2023. The recorded start, 2023-07-28 15:45 UTC, follows the listed resolution, 2023-07-26 16:30 UTC; computed duration is unknown. No cause or fix is shown.

Metabaseand MMetabase CloudUnable to create new Metabase cloud instances

Customers were unable to create new Metabase cloud trials or cloud instances in a critical incident affecting Metabase Store. It started at 2023-07-28 13:10 UTC and was resolved at 2023-07-29 10:21 UTC, lasting 21 h 10 min; no cause or fix was stated.

MMetabase Cloudand MetabaseUnable to create new database connections

Metabase Cloud customers could not add new databases or edit database connections during a major incident affecting Metabase Cloud Platform. It started at 2023-07-28 13:01 UTC and was resolved at 2023-07-29 09:06 UTC, lasting 20 h 4 min; no cause or specific fix was stated.

SSigma ComputingScheduled exports using image (PNG) files erroring

Sigma Computing reported a major (Severity 0) incident affecting all scheduled export types, with errors later described as intermittent across AWS, Azure and GCP. It started at 2023-07-27 13:29 UTC and was resolved at 2023-07-27 14:13 UTC, lasting 44 min. Sigma implemented a fix and said customers could resend impacted exports.

RootlySlack Outage (3rd Party) Impacting Rootly Slack App

A major Slack outage affected Rootly’s Slack app, including channel creation and notifications, from 2023-07-27 09:00 UTC to 2023-07-27 10:05 UTC (1 h 4 min). Rootly.com’s Web App remained operational for incident creation; Rootly’s services were fully restored after Slack restored functionality.

CCourierInbox messages degradation for versions lower than v2

CockroachDBServerless clients unable to access GCP Multi-Region Serverless Public Preview hosts

CockroachDB reported a major outage affecting connections to Serverless clusters on the GCP Multi-Region Serverless Public Preview host; other serverless clusters remained fully functional. The incident started at 2023-07-25 21:55 UTC and was resolved at 2023-07-25 23:07 UTC, lasting 1 h 12 min; no cause or specific fix was stated.

KeeperLogin Errors - US Production

Keeper reported critical US Production login errors affecting Keeper Admin Console (US) and Keeper Web Vault (US), from 2023-07-25 20:53 UTC to 2023-07-25 20:57 UTC (4 min). Errors followed a scheduled backend API software update and resolved when it completed; the underlying cause remained under investigation.

MattermostIncident with Community

Mattermost reported degraded availability for Community with major impact, starting at 2023-07-25 10:37 UTC and resolved at 2023-07-25 11:15 UTC; the computed duration was 37 min. A mitigation was applied at 11:10 UTC, but no specific cause was stated.

CCourierDelay in the messages sent out

LLogRocketLogRocket dashboard degraded performance

LogRocket reported a major degradation affecting Dashboard and Issues starting 2023-07-20 22:30 UTC, resolved at 2023-07-21 00:27 UTC after 1 h 56 min. Users could experience missing sourcemap uploads; data collection was unaffected. LogRocket identified the issue and implemented a fix, but stated no cause.

Ccrates.ioBroken download links

DDremioDatabase Connectivity

Dremio reported a major database connectivity incident affecting Dremio Cloud - US - Site Availability and Site Availability. It started and was resolved at 2023-07-18 22:42 UTC, with a recorded duration of 0 min; Dremio said it mitigated the connectivity issue and database connections recovered.

DDropbox SignSite Outage

Dropbox Sign reported a major site outage affecting signing, signature requests, document delivery, API callbacks, Dropbox Forms, fax services, and HubSpot and Salesforce integrations. It started at 2023-07-13 20:49 UTC and was resolved at 2023-07-13 20:53 UTC, with a computed duration of 3 min; no cause or fix was stated.

PostmanUpstream dependency issues for Bitbucket

Postman reported a major outage affecting Postman Integrations, with upstream dependency issues disrupting Integration APIs for Bitbucket. It started at 2023-07-13 04:51 UTC and was resolved at 2023-07-13 07:45 UTC, lasting 2 h 54 min.

MapboxDegraded Live Traffic Data.

Mapbox reported critical live traffic degradation starting 2023-07-13 00:55 UTC, lasting 2 h 17 min until resolution at 2023-07-13 03:12 UTC. Customers using Navigation APIs powered by Mapbox traffic could receive inaccurate ETAs, while traffic data customers could receive intermittent live updates in impacted regions. A fix was implemented at 07-13 01:53 UTC.

LLightstep (ServiceNow Cloud Observability)Lightstep experience major issues with web UI, query and ingest.

OpsgenieIntermittent errors during login for some customers

Opsgenie reported a major incident causing intermittent login errors for some customers, with EU and US Signup, Login & Authorization components affected. The incident started at 2023-07-06 09:03 UTC and was resolved at 2023-07-06 13:32 UTC, lasting 4 h 29 min; the vendor said it identified the root cause and mitigated the problem.

IimgixIssue with deploying new/updated Source configurations

IimgixElevated rendering errors

GustoMost users are experiencing errors within their Gusto account

Most Gusto users experienced account errors affecting Payroll, Benefits, and HR in a major outage starting 2023-06-30 17:52 UTC. Gusto implemented a fix by 2023-06-30 18:00 UTC and resolved the incident at 2023-06-30 19:00 UTC; it lasted 1 h 8 min.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site from 2023-06-26 16:10 UTC to 2023-06-26 16:25 UTC, lasting 15 min. A fix was implemented by 2023-06-26 16:20 UTC, and the incident was resolved.

MMailjet503 errors in the API

PostmanSeveral Postman services and features are degraded or not accessible

Postman reported a critical outage and degradation affecting Login, Monitors, and Platform on Browser and Desktop, caused by degraded internal identity services. The incident ran from 2023-06-23 04:58 UTC to 2023-06-23 08:07 UTC (3 h 9 min); all services returned to normal operating ranges before resolution.

DDremioEngine creation timeouts

Dremio Cloud Europe experienced engine creation timeout errors because engine startup was not properly recognized in the control plane; Dremio Cloud US was unaffected. The vendor marked the impact major, with the incident starting and resolving at 2023-06-22 14:00 UTC (computed duration: 0 min).

PyPI (Python Infrastructure)Object storage provider outage

MMailjetIntermittent access issues and slowness on the application

HHexPlatform availability issue

Hex reported a major platform availability outage affecting Login and Main site, starting at 2023-06-20 13:40 UTC and lasting 3 h 2 min. A fix was implemented by 2023-06-20 14:45 UTC, and the incident was resolved at 2023-06-20 16:43 UTC.

GustoSome International Contractors are experiencing onboarding and bank update errors.

Some Gusto International Contractors experienced onboarding and bank update errors in a major-impact incident affecting Payroll, Benefits, HR. It started at 2023-06-14 22:14 UTC and was resolved at 2023-06-15 01:06 UTC, lasting 2 h 51 min; no cause or specific fix was stated.

DagsterPartial outage of Dagster Cloud

Dagster Cloud had a vendor-rated critical partial outage affecting DNS routing, API and Dagster Cloud UI, starting 2023-06-14 17:44 UTC and lasting 16 min until resolution at 2023-06-14 18:00 UTC. DNS records were updated and propagating; users still unable to access Dagster Cloud were advised to clear their DNS cache.

GGrainAWS outage causing video processing delaysEvent

Grain reported a critical Recording Processing incident: an Amazon outage delayed video processing from 2023-06-13 20:11 UTC to 2023-06-14 02:39 UTC, lasting 6 h 27 min. Grain reported that the Amazon outage appeared resolved and the video backlog was processing, then confirmed all systems were operating correctly.

MapboxMaps API errorsEvent

Mapbox reported a critical degradation affecting offline map downloads, Mapbox Tiling Service, and live traffic generation and delivery, including delivery to Mapbox Speeds customers. The incident started at 2023-06-13 19:20 UTC and was resolved at 2023-06-13 23:20 UTC, lasting 4 h 0 min.

CCourierAn incident with Courier's hosting infrastructure has been identified and is impacting access to service.

GustoMost users are unable to get 2-step verification codes

Most Gusto users were unable to get 2-step verification codes in a critical outage affecting Payroll, Benefits, HR, and Two-factor authentication text messages. It started at 2023-06-08 14:39 UTC and was resolved at 2023-06-08 17:05 UTC, lasting 2 h 25 min; Gusto reported identifying the issue and implementing a fix.

MMailjetThe application is unreliable for some customers.

TTwingateadmin console and authentication incident

OpsgeniePartial Outage in Schedule API

Opsgenie reported a major partial outage of the Schedule API affecting some US Cloud customers and listing Configuration REST APIs and US - Configuration REST APIs as affected components. It ran from 2023-06-06 17:06 UTC to 2023-06-06 17:24 UTC, lasting 18 min; engineering fixed the problem and was monitoring the platform.