What vendors changed

HHexIntermittent 502s when accessing Hex

Hex reported major intermittent 502 errors affecting access to Main site and Single tenant stacks, starting 2024-06-20 19:18 UTC and resolved at 2024-06-20 20:32 UTC after 1 h 14 min. Hex said it was following a Cloudflare incident involving intermittent connectivity issues; no specific fix was stated.

TypeformTypeform Follow ups - Form notifications not being delivered

Typeform reported a major outage affecting Email Follow-ups: configured form notifications were not being delivered from 2024-06-20 18:20 UTC until resolution at 2024-06-21 19:37 UTC, lasting 25 h 16 min. A fix was implemented, pending notifications were delivered, and new form entry notifications were reported working during recovery.

DDropbox SignPartial Dropbox Sign Site Outage

Dropbox Sign reported a major partial site outage affecting API callbacks, document delivery, signature requests, signing, Dropbox Forms, and fax services. It started at 2024-06-17 23:17 UTC and was resolved at 2024-06-17 23:37 UTC, lasting 20 min; no cause or fix was stated.

LLightdashEmail not sending

Lightdash reported a major email outage affecting Lightdash Cloud (EU) and Lightdash Cloud (US), including signup verification and password resets, starting 2024-06-17 19:03 UTC and lasting 10 h 7 min. Lightdash attributed it to its email provider’s outage; emails were processing again at 06-17 20:50 UTC, and the incident was resolved at 2024-06-18 05:10 UTC.

AsanaElevated error rates

A major Asana outage left a majority of users unable to access Asana, affecting API and App components in Australia, EU, Japan and US. It ran from 2024-06-17 16:44 UTC to 2024-06-17 19:49 UTC (3 h 4 min). Asana attributed failures to load balancer resource saturation under high traffic; a fix was implemented at 06-17 16:52 UTC.

IimgixElevated rendering errors

npmFailures publishing, deprecating, and installing packages

GGrainRecorder scaling issues

Grain Recorder had a major degradation starting 2024-06-07 15:24 UTC, lasting 2 h 53 min until resolution at 2024-06-07 18:17 UTC. Recorder service scaling issues caused some bots to fail to join calls in a timely manner; Grain deployed a fix and reported no further issues.

LLightstep (ServiceNow Cloud Observability)Degraded spans queries

PusherStats collection experiencing an interruption

Pusher reported a critical interruption in Channels Stats Integrations: stats collection stopped at 2024-06-05 21:01 UTC, leaving customers without stats collection for 1 h 38 min. The vendor reported a fix at 2024-06-05 22:23 UTC and resolved the incident at 2024-06-05 22:39 UTC; no cause was stated.

PyPI (Python Infrastructure)Outage in python.org and us.pycon.org

NNordLayerElevated levels of connection issues

NordLayer reported major connectivity issues affecting Dedicated Servers, Private Gateways (Servers with dedicated IPs), Shared Gateways and Shared Servers, starting 2024-06-05 14:30 UTC. The incident lasted 22 h 42 min and was marked resolved at 2024-06-06 13:12 UTC, with a few isolated connectivity cases still being handled separately.

OpsgenieError responses across multiple Cloud products

Opsgenie reported a major outage affecting Cloud access, including EU and US mobile/web applications and signup, login and authorization. Recorded from 2024-06-03 23:11 UTC to 2024-06-04 00:32 UTC (1 h 21 min), it was resolved. The postmortem cites an infrastructure API Gateway change; its impact window differs from the recorded timestamps.

DagsterDagster Cloud UI Unavailable

Dagster Cloud UI was unavailable in a critical-impact incident starting 2024-06-03 21:56 UTC and resolved at 2024-06-03 22:01 UTC, with a reported duration of 4 min. Dagster attributed the loss of UI access to a load balancer configuration change and rolled it back.

CCensusBrief downtime for unscheduled database maintenance

Census reported a major outage affecting the Census Sync Management UI and Sync Engine during unscheduled emergency maintenance of a primary database. Users could see UI and API downtime; failed syncs would be automatically retried. The incident started at 2024-06-03 19:37 UTC and was resolved at 2024-06-03 19:42 UTC, with a reported duration of 4 min.

npmIntermittent failures installing scoped packages

HotjarSome session data not being ingested

Hotjar reported a critical degradation in which some session data was not being ingested, affecting session capture. It started at 2024-06-03 17:24 UTC and was resolved at 2024-06-05 09:22 UTC, lasting 39 h 57 min. Engineers deployed temporary session-capture changes and restored ingestion to regular levels while working with their cloud provider; no root cause was stated.

MMailjetLatency and login issues with Mailjet Application

SSigma ComputingUsers Unable To Contact Live Chat Support

Sigma Computing’s major live chat outage prevented users from contacting support through the Sigma application from 2024-05-29 15:17 UTC to 2024-05-29 17:23 UTC, lasting 2 h 6 min. Sigma directed customers to support@sigmacomputing.com during the outage and reported that its third-party chat service resolved the issue.

MMailjetLogin Issues with Mailjet Application

SSecureframeTrust Center pages not fully loading

Secureframe’s Trust Center pages were not fully loading in a vendor-rated critical incident affecting Trust. It started at 2024-05-24 22:36 UTC and lasted 16 min, resolving at 2024-05-24 22:52 UTC; Secureframe reported implementing a fix at 22:45 UTC.

RRudderStackData Ingestion affected for starter and growth users in EU region for 11 min

RudderStack reported a major Data Ingestion outage affecting starter and growth users in the EU region, starting at 2024-05-22 12:00 UTC and resolved at 2024-05-22 12:51 UTC. The computed incident duration was 50 min; the vendor provided no cause or fix details.

GGrainIssues with Meet calls

Grain Recorder’s Meet recordings failed to capture video and speaker change events in a major incident starting 2024-05-21 15:44 UTC. A fix was deployed by 05-22 00:30 UTC; Grain reported no further issues and marked it resolved at 2024-06-07 18:18 UTC, with a recorded duration of 17 days 2 h.

KeeperOkta SSO Logins

Keeper reported a major incident affecting some Okta SSO logins to Keeper Web Vault (US), from 2024-05-17 20:25 UTC to 2024-05-17 20:36 UTC (11 min). A backend software update caused login errors; DevOps reverted the change, resolving the incident, with a bug fix planned for a follow-up release.

CockroachDBGCP Networking Outage

A major Google Cloud Platform networking outage affected a select number of GCP Cockroach Cloud clusters from 2024-05-17 00:42 UTC to 2024-05-17 02:01 UTC, lasting 1 h 19 min. Cockroach Labs monitored the issue and reported that it was resolved.

PPandaDocDocument can not be downloaded as PDF files

PandaDoc reported a critical outage preventing documents from being downloaded as PDF files, affecting Uploading documents and US & Global - Uploading and downloading documents. It started at 2024-05-16 18:39 UTC and was resolved at 2024-05-16 19:57 UTC, lasting 1 h 18 min; PandaDoc confirmed normal operation but provided no specific cause or fix details.

HHexPlatform instability

Hex’s Main site experienced platform instability with major impact from 2024-05-15 15:24 UTC to 2024-05-15 16:13 UTC, lasting 48 min. A fix was implemented by 15:51 UTC, and the incident was resolved at 16:13 UTC.

DagsterSome Dagster Plus Serverless Runs Failing to Start

Some Dagster Plus Serverless runs failed to start with an "Agent detected that run worker failed." error during a major-impact incident from 2024-05-14 18:53 UTC to 2024-05-14 20:16 UTC, lasting 1 h 22 min. Dagster rolled out additional automatic retries; its cloud provider identified a causal change and began rolling it back. Dagster confirmed the fix was live.

LLightstep (ServiceNow Cloud Observability)Telemetry ingest severely degraded (spans, logs, metrics)

SStream (GetStream)High error rate on one of our Feed shards in the us-east region

IimgixAssets not displaying in Asset Manager

LLightstep (ServiceNow Cloud Observability)High failure rate in metric queries

TTwingateLogins with Github Not Working

CCloseCalling outage

Close reported a critical calling outage affecting Phone - Phone Service and Phone Service, starting 2024-04-22 14:53 UTC and resolved 2024-04-22 18:33 UTC, lasting 3 h 40 min. Close attributed the disruption to Twilio issues; Twilio implemented a fix, restoring baseline call functionality by 17:36 UTC.

CCensusSome users unable to log in to Census UI

Some users could not log in to the Census Sync Management UI, seeing "Verify Your Email" and "Something went wrong with your session" errors. Census rated the incident critical; it ran from 2024-04-19 16:23 UTC to 2024-04-19 17:22 UTC (59 min). A fix was implemented at 16:53 UTC, and the incident was resolved.

CCriblUser authentication and authorization to the cloud portal is degraded

Dagsterdagster.plus redirect breaks login for some organizations

A major incident affecting API and Dagster Cloud UI broke login and interactive web use for organizations whose firewalls allow-listed dagster.cloud, after erroneous redirects to dagster.plus. It ran from 2024-04-17 13:20 UTC to 2024-04-17 13:53 UTC (33 min); Dagster rolled back the redirects and implemented a fix, advising users with lingering login trouble to clear their browser cache.

KeeperVault API errors

Keeper reported a major incident involving Vault API errors affecting Keeper Admin Console and Keeper Web Vault in the EU and US. It started at 2024-04-15 10:05 UTC and was resolved at 2024-04-15 10:12 UTC, lasting 7 min; Keeper said it was working with AWS to address the issue.

DagsterSome Dagster Cloud API Requests failing with an Internal Server Error

Some Dagster Cloud API requests and a small number of runs failed with Internal Server Error exceptions due to a storage-engine timeout. The vendor rated the incident major; it ran from 2024-04-11 17:02 UTC to 2024-04-11 21:25 UTC, lasting 4 h 23 min. A fix was implemented and the incident resolved.

KeeperLogin errors - Resolved

Keeper reported a critical login outage affecting Keeper Admin Console (US) and Keeper Web Vault (US), starting 2024-04-11 12:55 UTC and resolved at 2024-04-11 13:12 UTC (17 min). Keeper attributed it to an underlying AWS database error that triggered a successfully handled failover; root-cause investigation with AWS continued.

SSecureframeUK Agents failing to report

Secureframe’s UK Agents failed to report in a major incident affecting Secureframe Agent from 2024-04-10 18:48 UTC to 2024-04-10 20:05 UTC, lasting 1 h 17 min. A fix was implemented and monitored before the incident was resolved.

CCensusSyncs not running

Census reported a critical Sync Engine outage in which syncs were not running, starting at 2024-04-09 23:32 UTC and lasting 3 h 9 min until resolution at 2024-04-10 02:41 UTC. A fix was implemented by 04-09 23:50 UTC, followed by monitoring.

OOmni AnalyticsDegraded Performance

Omni Analytics reported major degraded performance in EastUsa, with most queries failing to complete. The incident ran from 2024-04-09 18:44 UTC to 2024-04-10 01:04 UTC (6 h 19 min). Queries against a specific database type exhausted memory, repeatedly crashing querying until cancelled; Omni identified a fix for release that day and later marked the incident resolved.

OOmni AnalyticsSlow Queries

Omni Analytics reported major query degradation in EastUsa from 2024-04-08 16:55 UTC to 2024-04-08 17:05 UTC, lasting 10 min. Queries were slow or delayed because an information-schema query spike after cache expiration triggered a thundering herd, compounded by missing autoscaling logic; the incident was resolved, with autoscaling improvements planned.

PPipedream502 Bad Gateway on pipedream.com

Pipedream reported a major outage affecting pipedream.com, with elevated CPU causing 502 errors. It started at 2024-04-05 23:26 UTC and was resolved at 2024-04-05 23:28 UTC, lasting 2 min.

SSecureframeSecureframe Agent unable to connect

Secureframe reported a critical outage: Secureframe Agent could not connect to internal services, leaving it unable to operate and/or report. It lasted 20 h 22 min, from 2024-04-04 21:20 UTC to 2024-04-05 17:42 UTC. Secureframe implemented a fix and said it was working on an Agent update to reduce susceptibility to similar interruptions.

HotjarIssues with billing system

Hotjar reported a critical billing-system outage starting 2024-04-02 15:02 UTC, with access to checkout and invoices temporarily disabled during investigation. The incident was resolved at 2024-04-02 17:41 UTC; the recorded duration was 2 h 38 min.

RootlyIntermittent Failures with User Sign In and Incident Creation

Rootly reported a major incident causing intermittent sign-in and incident-creation failures across Slack, Web, and API; Rootly On-Call remained unaffected. It started at 2024-04-01 14:53 UTC and resolved at 2024-04-01 15:19 UTC, lasting 25 min. Rootly reported implementing a fix but did not disclose the cause; an RCA was promised to customers.

PPendoPendo login issue

Pendo reported a critical infrastructure issue affecting application login in Pendo UI, starting 2024-03-31 19:22 UTC and resolved at 2024-03-31 23:21 UTC, lasting 3 h 58 min. Fixes were reported at 03-31 20:39 UTC and 03-31 22:47 UTC, with renewed investigation between them.

JFrogJFrog Cloud Platform Unavailable For Some Customers on AWS US East

JFrog reported a major outage affecting some customers on AWS US East, with Artifactory, Distribution, Mission Control, Pipelines and Xray listed as affected. It started at 2024-03-28 16:38 UTC and was resolved at 2024-03-28 21:54 UTC, lasting 5 h 16 min; JFrog reported implementing a fix and confirmed all services operational.