What vendors changed

GustoMost users are unable to access their Gusto account

Most users were unable to access their Gusto accounts in a critical incident affecting Payroll, Benefits, and HR. It started at 2023-05-03 21:23 UTC and lasted 21 min, resolving at 2023-05-03 21:44 UTC; Gusto reported implementing a fix at 21:33 UTC.

LLightstep (ServiceNow Cloud Observability)intermittent UI unavailability issues

MezmoThe Web UI is not accessible

Mezmo reported a critical Web App outage: the Web UI was inaccessible and returned “failure to get a peer from the ring-balancer.” The incident record started at 2023-05-01 20:18 UTC and was resolved at 2023-05-01 20:27 UTC, lasting 9 min; the postmortem lists a different outage window.

PPandaDocPandaDoc performance: catalog CSV import

PandaDoc reported a major degradation affecting catalog CSV import and application performance, including smart content and email delivery. It started at 2023-04-27 16:04 UTC and was resolved at 2023-04-27 16:39 UTC; the recorded duration was 34 min. Performance was stabilized by 16:19 UTC, with no cause stated.

HHexTable display filter inconsistency

Hex reported a major table display filter inconsistency affecting Main site and Single tenant stacks, starting 2023-04-26 17:45 UTC and resolved at 2023-04-26 19:23 UTC, lasting 1 h 38 min. The vendor identified the issue and implemented a fix but provided no cause or fix details.

NNordLayerConnectivity issues with VPN servers in Sydney, Australia

NordLayer reported major connectivity issues affecting dedicated and shared VPN servers in Sydney, Australia, from 2023-04-18 06:43 UTC to 2023-04-18 07:33 UTC (49 min). The service provider implemented a fix, and NordLayer confirmed no further connectivity issues.

HHexPlatform availability issue

Hex reported a major platform availability issue affecting the Main site, starting at 2023-04-12 15:39 UTC and resolved at 2023-04-12 15:58 UTC. The incident lasted 18 min; no cause or specific fix was stated.

MMailjetLogin Issues with Mailjet Application

HHexPlatform availability issue

Hex reported a critical platform availability issue affecting the Main site from 2023-04-10 20:49 UTC to 2023-04-10 20:54 UTC, lasting 5 min. The incident was resolved; no cause or fix was stated.

GGrainZoom DNS Outage causing bots to fail

Grain reported a critical Grain Recorder outage starting 2023-04-08 11:52 UTC: Zoom’s DNS outage prevented recording bots from joining meetings, causing most bot recordings to fail. Grain recommended Zoom Cloud Recordings; Zoom resolved the DNS outage, with no further recorder failures observed as of 15:09 UTC. The incident closed at 2023-04-08 18:36 UTC, lasting 6 h 44 min.

PPandaDocPublic API is down

PandaDoc reported a critical Public API outage starting 2023-04-07 21:01 UTC; unresponsive API endpoints affected platform availability. The incident lasted 12 h 22 min and was resolved at 2023-04-08 09:23 UTC, after PandaDoc reported a fix and monitored application performance.

StatuspageIncreased error rate while SMS Subscription signups

Statuspage reported a major incident affecting SMS subscription signups, with increased errors on Notifications - SMS and SMS. It started at 2023-04-06 15:52 UTC and was resolved at 2023-04-06 19:17 UTC, lasting 3 h 24 min; the vendor described the issue as intermittent but stated no cause or fix.

PostmanUsers are not able sign-up

Postman reported a major outage affecting sign-ups on Postman Platform on Browser due to third-party library unavailability. It started at 2023-04-04 19:41 UTC and was resolved at 2023-04-04 20:03 UTC, lasting 21 min; Postman reported a fix at 04-04 19:55 UTC.

OpsgenieDelays in Android push notifications

Opsgenie reported a major incident delaying Android push notifications in the EU and US, starting 2023-04-04 14:30 UTC and resolved at 2023-04-04 18:16 UTC (3 h 45 min). The vendor attributed the delays to a Firebase error and confirmed Android push notifications were operational when Firebase’s incident was resolved.

PPandaDocPipedrive Integration is not Functioning Properly

PandaDoc’s Pipedrive integration stopped working in a critical incident from 2023-03-31 16:08 UTC to 2023-03-31 17:50 UTC, lasting 1 h 42 min. PandaDoc linked the issue to unexpected changes on the Pipedrive API side and reported functionality restored before marking the incident resolved.

CCourierMessage Status Delays

CCriblSome organizations have missing ACL rules in the Cloud Portal

SStream (GetStream)Elevated error rate for Feed apps in dublin region

CCriblSome organizations have invalid worker configuration

PPandaDocIssues with PandaDoc document creation, completion and download

PandaDoc reported a major outage affecting document creation, completion and download from 2023-03-20 17:15 UTC to 2023-03-20 18:20 UTC, lasting 1 h 4 min. PandaDoc said these functions were restored at 17:52 UTC and monitored performance before marking the incident resolved; no cause was stated.

PyPI (Python Infrastructure)python.org backend outage

PyPI (Python Infrastructure)PyPI Emails Not Sending

ShopifySome Merchants are experiencing issues accessing Storefronts and Shopify Admins

Shopify reported a critical outage affecting some merchants’ Storefronts, Checkout, Shopify Admins and Support/Help Center access, starting 2023-03-13 15:24 UTC and lasting 41 min until resolution at 2023-03-13 16:05 UTC. Checkouts, Storefronts and the Help Center recovered first; Shopify then reported implementing a fix for Admin access.

SSvixMost routes 5xx in the US region [resolved]

CCloseDelayed Webhook events

Close reported a major API degradation delaying webhook events from 2023-03-01 13:24 UTC to 2023-03-01 15:39 UTC, lasting 2 h 15 min. A fix was implemented by 14:13 UTC, and the event backlog was fully processed by resolution.

LLightstep (ServiceNow Cloud Observability)Elevated errors in alerts, API time series queries, dashboards, notebooks

CodaPartial outage of pack requests from clients

Coda reported a major Packs outage: some client-initiated pack requests failed from 2023-02-26 17:23 UTC to 2023-02-26 21:10 UTC, lasting 3 h 47 min. Coda suspected AWS CloudFront HTTP/2 handling and confirmed resolution after deploying a workaround.

TTeleportTeleport Cloud Regional Outages

GustoSome administrators are unable to process payroll

Some Gusto administrators were unable to process payroll in a major incident affecting Payroll, Benefits, HR, starting 2023-02-24 00:07 UTC and resolved at 2023-02-24 13:59 UTC (13 h 52 min). Gusto reported a fix at 00:31 UTC and advised customers to process payroll and use their account’s Help tab if they needed an expedite.

LLightstep (ServiceNow Cloud Observability)Not evaluating unified alerts

GustoMost administrators are experiencing errors processing contractor payments.

Gusto reported a major outage in which most administrators encountered errors processing contractor payments, from 2023-02-23 18:55 UTC to 2023-02-23 23:25 UTC, lasting 4 h 30 min. Gusto initially advised against processing contractor payments, then identified a fix and asked customers to review processed payments for correctness; the incident was resolved.

CCourierDelayed Automation Execution

LLightstep (ServiceNow Cloud Observability)Span Ingestion & Trace Assembly Degraded

SSecureframeSecureframe Not Reachable

Secureframe reported a critical outage affecting the API and Web Application, with customers unable to load part or all of Secureframe. It started at 2023-02-15 23:53 UTC and was resolved at 2023-02-16 00:21 UTC, lasting 28 min; no cause or fix was stated.

StatuspageSSO enabled private pages are facing authentication issues

Statuspage reported a major authentication outage affecting SSO-enabled private pages using Google Auth and SAML 2.0, from 2023-02-14 12:32 UTC to 2023-02-14 14:35 UTC (2 h 2 min). A fix was implemented by 2023-02-14 14:09 UTC, and the incident was resolved.

PPandaDocDocuments creation via API does not work

PandaDoc reported a major API outage that prevented document creation, starting at 2023-02-13 17:23 UTC and resolved at 2023-02-13 17:39 UTC, lasting 16 min. The vendor confirmed users could create documents via API again; no cause was stated.

MezmoSearches are running slowly

Mezmo reported major Search degradation starting 2023-02-10 18:11 UTC, resolved at 2023-02-10 18:14 UTC (computed duration: 2 min). Searches were slow or returned no results; no data was lost and ingestion continued. The postmortem lists an earlier start of 16:45 UTC; normal search speeds were restored.

PPandaDocUnable to work on the documents

PandaDoc reported a critical outage preventing users from working with documents, affecting the application and document sending/opening. It started at 2023-02-08 11:46 UTC and was resolved at 2023-02-08 12:49 UTC, lasting 1 h 2 min. The vendor reported document access restored at 12:33 UTC and monitored performance before closing the incident.

JJotformJotform Outage

Jotform reported a major outage starting at 2023-02-07 15:00 UTC and marked resolved at 2023-02-07 15:00 UTC, with a computed duration of 0 min. The notice did not specify affected features, a cause, or a fix.

MezmoIntermittent delays loading Web UI and running searches

Mezmo reported a major degradation affecting Livetail, Search and Web App from 2023-02-06 20:48 UTC to 2023-02-07 02:07 UTC, lasting 5 h 18 min. Searches were slow or returned no results, and the Web UI was intermittently unresponsive; all search data access was restored. No data was lost, and ingestion was not halted.

DDailyIssues connecting to calls.

MattermostCloudflare API issues are causing Cloud Sign-Ups to fail.

Mattermost reported a major outage causing cloud workspace sign-ups to fail due to Cloudflare API issues. It started at 2023-02-01 02:45 UTC and resolved at 2023-02-01 04:46 UTC, lasting 2 h 0 min. Mattermost reported that Cloudflare had implemented a fix and monitored the results before resolution.

TTabnineNo code completions

Tabnine reported a major outage affecting Cloud Advanced Completions: users could not get code completions. It started at 2023-01-31 17:45 UTC and was resolved at 2023-01-31 18:57 UTC, lasting 1 h 11 min. Tabnine identified corrupted configuration on its inference-engine gateway and restored service after deploying a configuration change to all servers.

DagsterDagster Cloud sensors and runs failing

Dagster Cloud reported a major API incident from 2023-01-27 19:14 UTC to 2023-01-27 20:50 UTC, lasting 1 h 36 min. A GraphQL-layer error caused sensors and runs using the k8s_job_executor to fail; a fix was implemented and the incident was resolved.

Ccrates.iostatic.crates.io not resolving

TTwingateTwingate Service Incident

LLightstep (ServiceNow Cloud Observability)Alert evaluation delays

FFireboltUI issue - reporting engines states

Firebolt reported a major UI incident affecting engine-state reporting across multiple accounts and engines, starting 2023-01-19 15:27 UTC and resolved 2023-01-19 18:43 UTC; the recorded duration was 3 h 15 min. Firebolt said the issue appeared UI-only, advised customers not to restart or stop engines, and applied a fix across all engines.

LLogRocketPerformance issues with primary search database

LogRocket reported a major primary search database degradation affecting session searches, metric charts, issues and alerting; session data collection was unaffected. It started at 2023-01-18 15:41 UTC and resolved at 2023-01-20 14:00 UTC, lasting 46 h 18 min. LogRocket applied a fix and processed event backlogs, reporting no further degradation after mitigation.

DDopplerUnable to sync secrets, webhooks, and activity log notifications

Doppler reported a major partial outage blocking secret syncs, webhooks, activity-log notifications, and API-client secret updates. Records list a start of 2023-01-16 20:37 UTC and resolution at 2023-01-16 19:11 UTC, while the postmortem reports 2023-01-16 19:11 UTC to 2023-01-16 20:04 UTC; duration is unknown.