What vendors changed

IimgixIssue with deploying new/updated Source configurations

IimgixElevated rendering errors

GustoMost users are experiencing errors within their Gusto account

Most Gusto users experienced account errors affecting Payroll, Benefits, and HR in a major outage starting 2023-06-30 17:52 UTC. Gusto implemented a fix by 2023-06-30 18:00 UTC and resolved the incident at 2023-06-30 19:00 UTC; it lasted 1 h 8 min.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site from 2023-06-26 16:10 UTC to 2023-06-26 16:25 UTC, lasting 15 min. A fix was implemented by 2023-06-26 16:20 UTC, and the incident was resolved.

MMailjet503 errors in the API

PostmanSeveral Postman services and features are degraded or not accessible

Postman reported a critical outage and degradation affecting Login, Monitors, and Platform on Browser and Desktop, caused by degraded internal identity services. The incident ran from 2023-06-23 04:58 UTC to 2023-06-23 08:07 UTC (3 h 9 min); all services returned to normal operating ranges before resolution.

DDremioEngine creation timeouts

Dremio Cloud Europe experienced engine creation timeout errors because engine startup was not properly recognized in the control plane; Dremio Cloud US was unaffected. The vendor marked the impact major, with the incident starting and resolving at 2023-06-22 14:00 UTC (computed duration: 0 min).

PyPI (Python Infrastructure)Object storage provider outage

MMailjetIntermittent access issues and slowness on the application

HHexPlatform availability issue

Hex reported a major platform availability outage affecting Login and Main site, starting at 2023-06-20 13:40 UTC and lasting 3 h 2 min. A fix was implemented by 2023-06-20 14:45 UTC, and the incident was resolved at 2023-06-20 16:43 UTC.

GustoSome International Contractors are experiencing onboarding and bank update errors.

Some Gusto International Contractors experienced onboarding and bank update errors in a major-impact incident affecting Payroll, Benefits, HR. It started at 2023-06-14 22:14 UTC and was resolved at 2023-06-15 01:06 UTC, lasting 2 h 51 min; no cause or specific fix was stated.

DagsterPartial outage of Dagster Cloud

Dagster Cloud had a vendor-rated critical partial outage affecting DNS routing, API and Dagster Cloud UI, starting 2023-06-14 17:44 UTC and lasting 16 min until resolution at 2023-06-14 18:00 UTC. DNS records were updated and propagating; users still unable to access Dagster Cloud were advised to clear their DNS cache.

GGrainAWS outage causing video processing delaysEvent

Grain reported a critical Recording Processing incident: an Amazon outage delayed video processing from 2023-06-13 20:11 UTC to 2023-06-14 02:39 UTC, lasting 6 h 27 min. Grain reported that the Amazon outage appeared resolved and the video backlog was processing, then confirmed all systems were operating correctly.

MapboxMaps API errorsEvent

Mapbox reported a critical degradation affecting offline map downloads, Mapbox Tiling Service, and live traffic generation and delivery, including delivery to Mapbox Speeds customers. The incident started at 2023-06-13 19:20 UTC and was resolved at 2023-06-13 23:20 UTC, lasting 4 h 0 min.

CCourierAn incident with Courier's hosting infrastructure has been identified and is impacting access to service.

GustoMost users are unable to get 2-step verification codes

Most Gusto users were unable to get 2-step verification codes in a critical outage affecting Payroll, Benefits, HR, and Two-factor authentication text messages. It started at 2023-06-08 14:39 UTC and was resolved at 2023-06-08 17:05 UTC, lasting 2 h 25 min; Gusto reported identifying the issue and implementing a fix.

MMailjetThe application is unreliable for some customers.

TTwingateadmin console and authentication incident

OpsgeniePartial Outage in Schedule API

Opsgenie reported a major partial outage of the Schedule API affecting some US Cloud customers and listing Configuration REST APIs and US - Configuration REST APIs as affected components. It ran from 2023-06-06 17:06 UTC to 2023-06-06 17:24 UTC, lasting 18 min; engineering fixed the problem and was monitoring the platform.

SSvixUS region is failing with 5xx errors an most API calls

LLightstep (ServiceNow Cloud Observability)Webapp is down

SStitchUser Interface Outage

Stitch reported a major outage of app.stitchdata.com that prevented users from logging in or using the interface, starting 2023-05-22 15:49 UTC and resolved at 2023-05-22 17:07 UTC (1 h 17 min). Stitch identified a networking change affecting front-end availability; service was restored by 16:32 UTC and remained stable during monitoring.

KeeperGovCloud SSO login 403 error - Resolved

Keeper reported a major incident affecting some GovCloud SSO logins with 403 errors, impacting the Admin Console and Web Vault. It started and was resolved at 2023-05-16 02:00 UTC; the recorded duration was 0 min. An overly aggressive WAF rule in Amazon Shield blocked requests; DevOps applied WAF rule policy changes to resolve it.

GustoSome users are experiencing site slowness

Gusto reported major site slowness affecting some users across API, Payroll, Benefits, and HR. The incident started at 2023-05-11 20:21 UTC and was resolved at 2023-05-11 20:55 UTC, lasting 34 min; no cause or fix was stated.

CCourierIssue affecting sends

GGrainApplication Outage

Grain reported a critical outage starting 2023-05-10 17:12 UTC that lasted 1 h 1 min, with all functionality offline due to database issues, affecting Grain Recorder, Grain Web App and Recording Processing. Services were gradually re-enabled, and full functionality was restored at 2023-05-10 18:13 UTC.

LLightdashLightdash Cloud instability (slow response times)

Lightdash Cloud (US) experienced major API degradation affecting all users, with actions taking minutes or not executing. It started at 2023-05-09 13:48 UTC and was resolved at 2023-05-09 15:33 UTC; the computed duration was 1 h 44 min. Lightdash reported increasing server count and size as its initial response.

SStream (GetStream)Realtime connections outage

GustoUsers are experiencing site slowness

Gusto reported major site slowness affecting API, Payroll, Benefits, and HR starting at 2023-05-04 22:33 UTC. The incident lasted 12 min and was resolved at 2023-05-04 22:45 UTC; no cause or fix was stated.

CCloseSMS provider issues

Close reported a major SMS outage affecting Phone - SMS (primary) and SMS, attributed to issues at telephony provider Twilio. It started at 2023-05-04 15:05 UTC and was resolved at 2023-05-04 17:42 UTC, lasting 2 h 36 min.

CCriblCribl cloud login erroring with "The provided redirect_uri is not in the list of allowed callback URLs."

TTeleportSome clusters failing to restart following Tenant Upgrade

DagsterReports of jobs not starting

Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution.

GustoMost users are unable to access their Gusto account

Most users were unable to access their Gusto accounts in a critical incident affecting Payroll, Benefits, and HR. It started at 2023-05-03 21:23 UTC and lasted 21 min, resolving at 2023-05-03 21:44 UTC; Gusto reported implementing a fix at 21:33 UTC.

LLightstep (ServiceNow Cloud Observability)intermittent UI unavailability issues

MezmoThe Web UI is not accessible

Mezmo reported a critical Web App outage: the Web UI was inaccessible and returned “failure to get a peer from the ring-balancer.” The incident record started at 2023-05-01 20:18 UTC and was resolved at 2023-05-01 20:27 UTC, lasting 9 min; the postmortem lists a different outage window.

PPandaDocPandaDoc performance: catalog CSV import

PandaDoc reported a major degradation affecting catalog CSV import and application performance, including smart content and email delivery. It started at 2023-04-27 16:04 UTC and was resolved at 2023-04-27 16:39 UTC; the recorded duration was 34 min. Performance was stabilized by 16:19 UTC, with no cause stated.

HHexTable display filter inconsistency

Hex reported a major table display filter inconsistency affecting Main site and Single tenant stacks, starting 2023-04-26 17:45 UTC and resolved at 2023-04-26 19:23 UTC, lasting 1 h 38 min. The vendor identified the issue and implemented a fix but provided no cause or fix details.

NNordLayerConnectivity issues with VPN servers in Sydney, Australia

NordLayer reported major connectivity issues affecting dedicated and shared VPN servers in Sydney, Australia, from 2023-04-18 06:43 UTC to 2023-04-18 07:33 UTC (49 min). The service provider implemented a fix, and NordLayer confirmed no further connectivity issues.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site, starting at 2023-04-12 15:39 UTC and resolved at 2023-04-12 15:58 UTC. The incident lasted 18 min; no cause or specific fix was stated.

MMailjetLogin Issues with Mailjet Application

HHexPlatform availability issue

Hex reported a critical platform availability issue affecting the Main site from 2023-04-10 20:49 UTC to 2023-04-10 20:54 UTC, lasting 5 min. The incident was resolved; no cause or fix was stated.

GGrainZoom DNS Outage causing bots to fail

Grain reported a critical Grain Recorder outage starting 2023-04-08 11:52 UTC: Zoom’s DNS outage prevented recording bots from joining meetings, causing most bot recordings to fail. Grain recommended Zoom Cloud Recordings; Zoom resolved the DNS outage, with no further recorder failures observed as of 15:09 UTC. The incident closed at 2023-04-08 18:36 UTC, lasting 6 h 44 min.

PPandaDocPublic API is down

PandaDoc reported a critical Public API outage starting 2023-04-07 21:01 UTC; unresponsive API endpoints affected platform availability. The incident lasted 12 h 22 min and was resolved at 2023-04-08 09:23 UTC, after PandaDoc reported a fix and monitored application performance.

StatuspageIncreased error rate while SMS Subscription signups

Statuspage reported a major incident affecting SMS subscription signups, with increased errors on Notifications - SMS and SMS. It started at 2023-04-06 15:52 UTC and was resolved at 2023-04-06 19:17 UTC, lasting 3 h 24 min; the vendor described the issue as intermittent but stated no cause or fix.

PostmanUsers are not able sign-up

Postman reported a major outage affecting sign-ups on Postman Platform on Browser due to third-party library unavailability. It started at 2023-04-04 19:41 UTC and was resolved at 2023-04-04 20:03 UTC, lasting 21 min; Postman reported a fix at 04-04 19:55 UTC.

OpsgenieDelays in Android push notifications

Opsgenie reported a major incident delaying Android push notifications in the EU and US, starting 2023-04-04 14:30 UTC and resolved at 2023-04-04 18:16 UTC (3 h 45 min). The vendor attributed the delays to a Firebase error and confirmed Android push notifications were operational when Firebase’s incident was resolved.

PPandaDocPipedrive Integration is not Functioning Properly

PandaDoc’s Pipedrive integration stopped working in a critical incident from 2023-03-31 16:08 UTC to 2023-03-31 17:50 UTC, lasting 1 h 42 min. PandaDoc linked the issue to unexpected changes on the Pipedrive API side and reported functionality restored before marking the incident resolved.

CCourierMessage Status Delays

CCriblSome organizations have missing ACL rules in the Cloud Portal