What vendors changed

OpsgeniePartial Outage in Schedule API

Opsgenie reported a major partial outage of the Schedule API affecting some US Cloud customers and listing Configuration REST APIs and US - Configuration REST APIs as affected components. It ran from 2023-06-06 17:06 UTC to 2023-06-06 17:24 UTC, lasting 18 min; engineering fixed the problem and was monitoring the platform.

SSvixUS region is failing with 5xx errors an most API calls

LLightstep (ServiceNow Cloud Observability)Webapp is down

SStitchUser Interface Outage

Stitch reported a major outage of app.stitchdata.com that prevented users from logging in or using the interface, starting 2023-05-22 15:49 UTC and resolved at 2023-05-22 17:07 UTC (1 h 17 min). Stitch identified a networking change affecting front-end availability; service was restored by 16:32 UTC and remained stable during monitoring.

KeeperGovCloud SSO login 403 error - Resolved

Keeper reported a major incident affecting some GovCloud SSO logins with 403 errors, impacting the Admin Console and Web Vault. It started and was resolved at 2023-05-16 02:00 UTC; the recorded duration was 0 min. An overly aggressive WAF rule in Amazon Shield blocked requests; DevOps applied WAF rule policy changes to resolve it.

GustoSome users are experiencing site slowness

Gusto reported major site slowness affecting some users across API, Payroll, Benefits, and HR. The incident started at 2023-05-11 20:21 UTC and was resolved at 2023-05-11 20:55 UTC, lasting 34 min; no cause or fix was stated.

CCourierIssue affecting sends

GGrainApplication Outage

Grain reported a critical outage starting 2023-05-10 17:12 UTC that lasted 1 h 1 min, with all functionality offline due to database issues, affecting Grain Recorder, Grain Web App and Recording Processing. Services were gradually re-enabled, and full functionality was restored at 2023-05-10 18:13 UTC.

LLightdashLightdash Cloud instability (slow response times)

Lightdash Cloud (US) experienced major API degradation affecting all users, with actions taking minutes or not executing. It started at 2023-05-09 13:48 UTC and was resolved at 2023-05-09 15:33 UTC; the computed duration was 1 h 44 min. Lightdash reported increasing server count and size as its initial response.

SStream (GetStream)Realtime connections outage

GustoUsers are experiencing site slowness

Gusto reported major site slowness affecting API, Payroll, Benefits, and HR starting at 2023-05-04 22:33 UTC. The incident lasted 12 min and was resolved at 2023-05-04 22:45 UTC; no cause or fix was stated.

CCloseSMS provider issues

Close reported a major SMS outage affecting Phone - SMS (primary) and SMS, attributed to issues at telephony provider Twilio. It started at 2023-05-04 15:05 UTC and was resolved at 2023-05-04 17:42 UTC, lasting 2 h 36 min.

CCriblCribl cloud login erroring with "The provided redirect_uri is not in the list of allowed callback URLs."

TTeleportSome clusters failing to restart following Tenant Upgrade

DagsterReports of jobs not starting

Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution.

GustoMost users are unable to access their Gusto account

Most users were unable to access their Gusto accounts in a critical incident affecting Payroll, Benefits, and HR. It started at 2023-05-03 21:23 UTC and lasted 21 min, resolving at 2023-05-03 21:44 UTC; Gusto reported implementing a fix at 21:33 UTC.

LLightstep (ServiceNow Cloud Observability)intermittent UI unavailability issues

MezmoThe Web UI is not accessible

Mezmo reported a critical Web App outage: the Web UI was inaccessible and returned “failure to get a peer from the ring-balancer.” The incident record started at 2023-05-01 20:18 UTC and was resolved at 2023-05-01 20:27 UTC, lasting 9 min; the postmortem lists a different outage window.

PPandaDocPandaDoc performance: catalog CSV import

PandaDoc reported a major degradation affecting catalog CSV import and application performance, including smart content and email delivery. It started at 2023-04-27 16:04 UTC and was resolved at 2023-04-27 16:39 UTC; the recorded duration was 34 min. Performance was stabilized by 16:19 UTC, with no cause stated.

HHexTable display filter inconsistency

Hex reported a major table display filter inconsistency affecting Main site and Single tenant stacks, starting 2023-04-26 17:45 UTC and resolved at 2023-04-26 19:23 UTC, lasting 1 h 38 min. The vendor identified the issue and implemented a fix but provided no cause or fix details.

NNordLayerConnectivity issues with VPN servers in Sydney, Australia

NordLayer reported major connectivity issues affecting dedicated and shared VPN servers in Sydney, Australia, from 2023-04-18 06:43 UTC to 2023-04-18 07:33 UTC (49 min). The service provider implemented a fix, and NordLayer confirmed no further connectivity issues.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site, starting at 2023-04-12 15:39 UTC and resolved at 2023-04-12 15:58 UTC. The incident lasted 18 min; no cause or specific fix was stated.

MMailjetLogin Issues with Mailjet Application

HHexPlatform availability issue

Hex reported a critical platform availability issue affecting the Main site from 2023-04-10 20:49 UTC to 2023-04-10 20:54 UTC, lasting 5 min. The incident was resolved; no cause or fix was stated.

GGrainZoom DNS Outage causing bots to fail

Grain reported a critical Grain Recorder outage starting 2023-04-08 11:52 UTC: Zoom’s DNS outage prevented recording bots from joining meetings, causing most bot recordings to fail. Grain recommended Zoom Cloud Recordings; Zoom resolved the DNS outage, with no further recorder failures observed as of 15:09 UTC. The incident closed at 2023-04-08 18:36 UTC, lasting 6 h 44 min.

PPandaDocPublic API is down

PandaDoc reported a critical Public API outage starting 2023-04-07 21:01 UTC; unresponsive API endpoints affected platform availability. The incident lasted 12 h 22 min and was resolved at 2023-04-08 09:23 UTC, after PandaDoc reported a fix and monitored application performance.

StatuspageIncreased error rate while SMS Subscription signups

Statuspage reported a major incident affecting SMS subscription signups, with increased errors on Notifications - SMS and SMS. It started at 2023-04-06 15:52 UTC and was resolved at 2023-04-06 19:17 UTC, lasting 3 h 24 min; the vendor described the issue as intermittent but stated no cause or fix.

PostmanUsers are not able sign-up

Postman reported a major outage affecting sign-ups on Postman Platform on Browser due to third-party library unavailability. It started at 2023-04-04 19:41 UTC and was resolved at 2023-04-04 20:03 UTC, lasting 21 min; Postman reported a fix at 04-04 19:55 UTC.

OpsgenieDelays in Android push notifications

Opsgenie reported a major incident delaying Android push notifications in the EU and US, starting 2023-04-04 14:30 UTC and resolved at 2023-04-04 18:16 UTC (3 h 45 min). The vendor attributed the delays to a Firebase error and confirmed Android push notifications were operational when Firebase’s incident was resolved.

PPandaDocPipedrive Integration is not Functioning Properly

PandaDoc’s Pipedrive integration stopped working in a critical incident from 2023-03-31 16:08 UTC to 2023-03-31 17:50 UTC, lasting 1 h 42 min. PandaDoc linked the issue to unexpected changes on the Pipedrive API side and reported functionality restored before marking the incident resolved.

CCourierMessage Status Delays

CCriblSome organizations have missing ACL rules in the Cloud Portal

SStream (GetStream)Elevated error rate for Feed apps in dublin region

CCriblSome organizations have invalid worker configuration

PPandaDocIssues with PandaDoc document creation, completion and download

PandaDoc reported a major outage affecting document creation, completion and download from 2023-03-20 17:15 UTC to 2023-03-20 18:20 UTC, lasting 1 h 4 min. PandaDoc said these functions were restored at 17:52 UTC and monitored performance before marking the incident resolved; no cause was stated.

PyPI (Python Infrastructure)python.org backend outage

PyPI (Python Infrastructure)PyPI Emails Not Sending

ShopifySome Merchants are experiencing issues accessing Storefronts and Shopify Admins

Shopify reported a critical outage affecting some merchants’ Storefronts, Checkout, Shopify Admins and Support/Help Center access, starting 2023-03-13 15:24 UTC and lasting 41 min until resolution at 2023-03-13 16:05 UTC. Checkouts, Storefronts and the Help Center recovered first; Shopify then reported implementing a fix for Admin access.

SSvixMost routes 5xx in the US region [resolved]

CCloseDelayed Webhook events

Close reported a major API degradation delaying webhook events from 2023-03-01 13:24 UTC to 2023-03-01 15:39 UTC, lasting 2 h 15 min. A fix was implemented by 14:13 UTC, and the event backlog was fully processed by resolution.

LLightstep (ServiceNow Cloud Observability)Elevated errors in alerts, API time series queries, dashboards, notebooks

CodaPartial outage of pack requests from clients

Coda reported a major Packs outage: some client-initiated pack requests failed from 2023-02-26 17:23 UTC to 2023-02-26 21:10 UTC, lasting 3 h 47 min. Coda suspected AWS CloudFront HTTP/2 handling and confirmed resolution after deploying a workaround.

TTeleportTeleport Cloud Regional Outages

GustoSome administrators are unable to process payroll

Some Gusto administrators were unable to process payroll in a major incident affecting Payroll, Benefits, HR, starting 2023-02-24 00:07 UTC and resolved at 2023-02-24 13:59 UTC (13 h 52 min). Gusto reported a fix at 00:31 UTC and advised customers to process payroll and use their account’s Help tab if they needed an expedite.

LLightstep (ServiceNow Cloud Observability)Not evaluating unified alerts

GustoMost administrators are experiencing errors processing contractor payments.

Gusto reported a major outage in which most administrators encountered errors processing contractor payments, from 2023-02-23 18:55 UTC to 2023-02-23 23:25 UTC, lasting 4 h 30 min. Gusto initially advised against processing contractor payments, then identified a fix and asked customers to review processed payments for correctness; the incident was resolved.

CCourierDelayed Automation Execution

LLightstep (ServiceNow Cloud Observability)Span Ingestion & Trace Assembly Degraded

SSecureframeSecureframe Not Reachable

Secureframe reported a critical outage affecting the API and Web Application, with customers unable to load part or all of Secureframe. It started at 2023-02-15 23:53 UTC and was resolved at 2023-02-16 00:21 UTC, lasting 28 min; no cause or fix was stated.

StatuspageSSO enabled private pages are facing authentication issues

Statuspage reported a major authentication outage affecting SSO-enabled private pages using Google Auth and SAML 2.0, from 2023-02-14 12:32 UTC to 2023-02-14 14:35 UTC (2 h 2 min). A fix was implemented by 2023-02-14 14:09 UTC, and the incident was resolved.