What vendors changed

CCourierInbox messages degradation for versions lower than v2

CockroachDBServerless clients unable to access GCP Multi-Region Serverless Public Preview hosts

CockroachDB reported a major outage affecting connections to Serverless clusters on the GCP Multi-Region Serverless Public Preview host; other serverless clusters remained fully functional. The incident started at 2023-07-25 21:55 UTC and was resolved at 2023-07-25 23:07 UTC, lasting 1 h 12 min; no cause or specific fix was stated.

KeeperLogin Errors - US Production

Keeper reported critical US Production login errors affecting Keeper Admin Console (US) and Keeper Web Vault (US), from 2023-07-25 20:53 UTC to 2023-07-25 20:57 UTC (4 min). Errors followed a scheduled backend API software update and resolved when it completed; the underlying cause remained under investigation.

MattermostIncident with Community

Mattermost reported degraded availability for Community with major impact, starting at 2023-07-25 10:37 UTC and resolved at 2023-07-25 11:15 UTC; the computed duration was 37 min. A mitigation was applied at 11:10 UTC, but no specific cause was stated.

CCourierDelay in the messages sent out

LLogRocketLogRocket dashboard degraded performance

LogRocket reported a major degradation affecting Dashboard and Issues starting 2023-07-20 22:30 UTC, resolved at 2023-07-21 00:27 UTC after 1 h 56 min. Users could experience missing sourcemap uploads; data collection was unaffected. LogRocket identified the issue and implemented a fix, but stated no cause.

Ccrates.ioBroken download links

DDremioDatabase Connectivity

Dremio reported a major database connectivity incident affecting Dremio Cloud - US - Site Availability and Site Availability. It started and was resolved at 2023-07-18 22:42 UTC, with a recorded duration of 0 min; Dremio said it mitigated the connectivity issue and database connections recovered.

DDropbox SignSite Outage

Dropbox Sign reported a major site outage affecting signing, signature requests, document delivery, API callbacks, Dropbox Forms, fax services, and HubSpot and Salesforce integrations. It started at 2023-07-13 20:49 UTC and was resolved at 2023-07-13 20:53 UTC, with a computed duration of 3 min; no cause or fix was stated.

PostmanUpstream dependency issues for Bitbucket

Postman reported a major outage affecting Postman Integrations, with upstream dependency issues disrupting Integration APIs for Bitbucket. It started at 2023-07-13 04:51 UTC and was resolved at 2023-07-13 07:45 UTC, lasting 2 h 54 min.

MapboxDegraded Live Traffic Data.

Mapbox reported critical live traffic degradation starting 2023-07-13 00:55 UTC, lasting 2 h 17 min until resolution at 2023-07-13 03:12 UTC. Customers using Navigation APIs powered by Mapbox traffic could receive inaccurate ETAs, while traffic data customers could receive intermittent live updates in impacted regions. A fix was implemented at 07-13 01:53 UTC.

LLightstep (ServiceNow Cloud Observability)Lightstep experience major issues with web UI, query and ingest.

OpsgenieIntermittent errors during login for some customers

Opsgenie reported a major incident causing intermittent login errors for some customers, with EU and US Signup, Login & Authorization components affected. The incident started at 2023-07-06 09:03 UTC and was resolved at 2023-07-06 13:32 UTC, lasting 4 h 29 min; the vendor said it identified the root cause and mitigated the problem.

IimgixIssue with deploying new/updated Source configurations

IimgixElevated rendering errors

GustoMost users are experiencing errors within their Gusto account

Most Gusto users experienced account errors affecting Payroll, Benefits, and HR in a major outage starting 2023-06-30 17:52 UTC. Gusto implemented a fix by 2023-06-30 18:00 UTC and resolved the incident at 2023-06-30 19:00 UTC; it lasted 1 h 8 min.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site from 2023-06-26 16:10 UTC to 2023-06-26 16:25 UTC, lasting 15 min. A fix was implemented by 2023-06-26 16:20 UTC, and the incident was resolved.

MMailjet503 errors in the API

PostmanSeveral Postman services and features are degraded or not accessible

Postman reported a critical outage and degradation affecting Login, Monitors, and Platform on Browser and Desktop, caused by degraded internal identity services. The incident ran from 2023-06-23 04:58 UTC to 2023-06-23 08:07 UTC (3 h 9 min); all services returned to normal operating ranges before resolution.

DDremioEngine creation timeouts

Dremio Cloud Europe experienced engine creation timeout errors because engine startup was not properly recognized in the control plane; Dremio Cloud US was unaffected. The vendor marked the impact major, with the incident starting and resolving at 2023-06-22 14:00 UTC (computed duration: 0 min).

PyPI (Python Infrastructure)Object storage provider outage

MMailjetIntermittent access issues and slowness on the application

HHexPlatform availability issue

Hex reported a major platform availability outage affecting Login and Main site, starting at 2023-06-20 13:40 UTC and lasting 3 h 2 min. A fix was implemented by 2023-06-20 14:45 UTC, and the incident was resolved at 2023-06-20 16:43 UTC.

GustoSome International Contractors are experiencing onboarding and bank update errors.

Some Gusto International Contractors experienced onboarding and bank update errors in a major-impact incident affecting Payroll, Benefits, HR. It started at 2023-06-14 22:14 UTC and was resolved at 2023-06-15 01:06 UTC, lasting 2 h 51 min; no cause or specific fix was stated.

DagsterPartial outage of Dagster Cloud

Dagster Cloud had a vendor-rated critical partial outage affecting DNS routing, API and Dagster Cloud UI, starting 2023-06-14 17:44 UTC and lasting 16 min until resolution at 2023-06-14 18:00 UTC. DNS records were updated and propagating; users still unable to access Dagster Cloud were advised to clear their DNS cache.

GGrainAWS outage causing video processing delaysEvent

Grain reported a critical Recording Processing incident: an Amazon outage delayed video processing from 2023-06-13 20:11 UTC to 2023-06-14 02:39 UTC, lasting 6 h 27 min. Grain reported that the Amazon outage appeared resolved and the video backlog was processing, then confirmed all systems were operating correctly.

MapboxMaps API errorsEvent

Mapbox reported a critical degradation affecting offline map downloads, Mapbox Tiling Service, and live traffic generation and delivery, including delivery to Mapbox Speeds customers. The incident started at 2023-06-13 19:20 UTC and was resolved at 2023-06-13 23:20 UTC, lasting 4 h 0 min.

CCourierAn incident with Courier's hosting infrastructure has been identified and is impacting access to service.

GustoMost users are unable to get 2-step verification codes

Most Gusto users were unable to get 2-step verification codes in a critical outage affecting Payroll, Benefits, HR, and Two-factor authentication text messages. It started at 2023-06-08 14:39 UTC and was resolved at 2023-06-08 17:05 UTC, lasting 2 h 25 min; Gusto reported identifying the issue and implementing a fix.

MMailjetThe application is unreliable for some customers.

TTwingateadmin console and authentication incident

OpsgeniePartial Outage in Schedule API

Opsgenie reported a major partial outage of the Schedule API affecting some US Cloud customers and listing Configuration REST APIs and US - Configuration REST APIs as affected components. It ran from 2023-06-06 17:06 UTC to 2023-06-06 17:24 UTC, lasting 18 min; engineering fixed the problem and was monitoring the platform.

SSvixUS region is failing with 5xx errors an most API calls

LLightstep (ServiceNow Cloud Observability)Webapp is down

SStitchUser Interface Outage

Stitch reported a major outage of app.stitchdata.com that prevented users from logging in or using the interface, starting 2023-05-22 15:49 UTC and resolved at 2023-05-22 17:07 UTC (1 h 17 min). Stitch identified a networking change affecting front-end availability; service was restored by 16:32 UTC and remained stable during monitoring.

KeeperGovCloud SSO login 403 error - Resolved

Keeper reported a major incident affecting some GovCloud SSO logins with 403 errors, impacting the Admin Console and Web Vault. It started and was resolved at 2023-05-16 02:00 UTC; the recorded duration was 0 min. An overly aggressive WAF rule in Amazon Shield blocked requests; DevOps applied WAF rule policy changes to resolve it.

GustoSome users are experiencing site slowness

Gusto reported major site slowness affecting some users across API, Payroll, Benefits, and HR. The incident started at 2023-05-11 20:21 UTC and was resolved at 2023-05-11 20:55 UTC, lasting 34 min; no cause or fix was stated.

CCourierIssue affecting sends

GGrainApplication Outage

Grain reported a critical outage starting 2023-05-10 17:12 UTC that lasted 1 h 1 min, with all functionality offline due to database issues, affecting Grain Recorder, Grain Web App and Recording Processing. Services were gradually re-enabled, and full functionality was restored at 2023-05-10 18:13 UTC.

LLightdashLightdash Cloud instability (slow response times)

Lightdash Cloud (US) experienced major API degradation affecting all users, with actions taking minutes or not executing. It started at 2023-05-09 13:48 UTC and was resolved at 2023-05-09 15:33 UTC; the computed duration was 1 h 44 min. Lightdash reported increasing server count and size as its initial response.

SStream (GetStream)Realtime connections outage

GustoUsers are experiencing site slowness

Gusto reported major site slowness affecting API, Payroll, Benefits, and HR starting at 2023-05-04 22:33 UTC. The incident lasted 12 min and was resolved at 2023-05-04 22:45 UTC; no cause or fix was stated.

CCloseSMS provider issues

Close reported a major SMS outage affecting Phone - SMS (primary) and SMS, attributed to issues at telephony provider Twilio. It started at 2023-05-04 15:05 UTC and was resolved at 2023-05-04 17:42 UTC, lasting 2 h 36 min.

CCriblCribl cloud login erroring with "The provided redirect_uri is not in the list of allowed callback URLs."

TTeleportSome clusters failing to restart following Tenant Upgrade

DagsterReports of jobs not starting

Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution.

GustoMost users are unable to access their Gusto account

Most users were unable to access their Gusto accounts in a critical incident affecting Payroll, Benefits, and HR. It started at 2023-05-03 21:23 UTC and lasted 21 min, resolving at 2023-05-03 21:44 UTC; Gusto reported implementing a fix at 21:33 UTC.

LLightstep (ServiceNow Cloud Observability)intermittent UI unavailability issues

MezmoThe Web UI is not accessible

Mezmo reported a critical Web App outage: the Web UI was inaccessible and returned “failure to get a peer from the ring-balancer.” The incident record started at 2023-05-01 20:18 UTC and was resolved at 2023-05-01 20:27 UTC, lasting 9 min; the postmortem lists a different outage window.

PPandaDocPandaDoc performance: catalog CSV import

PandaDoc reported a major degradation affecting catalog CSV import and application performance, including smart content and email delivery. It started at 2023-04-27 16:04 UTC and was resolved at 2023-04-27 16:39 UTC; the recorded duration was 34 min. Performance was stabilized by 16:19 UTC, with no cause stated.