What vendors changed

SingleStoreAWS outage in Stockholm region causing connectivity issuesEvent

PrismaAccelerate elevated 500 responses

PPipedreamElevated Error Rates

Pipedream reported major elevated error rates affecting HTTP and Timer event sources and workflows, including backend components. The incident started at 2025-02-12 04:41 UTC and was resolved at 2025-02-12 06:44 UTC, lasting 2 h 2 min; no cause or specific fix was stated.

CCloseDegraded Performance

Close reported a major incident affecting API, Application UI, and email/calendar syncing from 2025-02-11 22:31 UTC to 2025-02-11 23:34 UTC, lasting 1 h 2 min. Users experienced degraded performance and elevated errors; Close rolled out a fix, with UI/API performance recovering before delayed syncing was fully resolved.

GrammarlyGrammarly is experiencing temporary service interruption.

Grammarly reported a critical outage affecting Account Management, Grammarly Editor, Grammarly for Mac, Grammarly for Windows, Grammarly's Browser Extension, and Login; products were temporarily unavailable. It started at 2025-02-11 20:39 UTC and lasted 1 h 47 min, resolving at 2025-02-11 22:26 UTC. A fix was implemented at 21:18 UTC; no cause was stated.

KeeperLogin errors in US region

Keeper reported a major US-region login outage affecting Keeper Admin Console (US) and Keeper Web Vault (US), starting 2025-02-10 17:30 UTC and resolved at 2025-02-10 17:49 UTC (19 min). The postmortem attributed failed new login sessions to a database lock in the primary writer instance and said the issue resolved without intervention; a backend update was planned.

DDremioQueries stuck in metadata retrieval in Dremio Cloud (EU)

Dremio Cloud (EU) queries stalled at metadata retrieval during a major incident from 2025-02-10 10:02 UTC to 2025-02-10 17:55 UTC (7 h 53 min), now resolved. Dremio reported upstream timeouts and queuing in a core indexing service, mitigated by throttling; initial analysis suggested a recent underlying infrastructure change on Google may have triggered the failure.

SSambaNovaDeepSeek-R1-Distill-Llama-70B Down during feature rollout

DagsterIssues with global op concurrency keys

Dagster reported a major API and Dagster Cloud UI degradation from 2025-02-07 15:37 UTC to 2025-02-07 16:11 UTC (34 min): some runs with concurrency keys lacking explicit configuration or a default limit were blocked, and concurrency settings had UI issues. Fixes restored progress, but previously blocked runs might still require manual termination.

HHexInstability in project loads

Hex reported major instability in project loads affecting the Main site, starting 2025-02-06 19:39 UTC and resolved at 2025-02-06 21:29 UTC, lasting 1 h 49 min. Hex stated that a fix had been implemented and was being monitored at the start of the incident.

CCensusSome syncs failing with "Missing corresponding dataset file for prefix: record_deletes.0"

Census reported a major Sync Engine incident starting 2025-02-05 22:38 UTC: some syncs failed with "Missing corresponding dataset file for prefix: record_deletes.0" after a code change introduced a bug. A revert deployed at 22:56 UTC; the incident was resolved at 2025-02-05 23:30 UTC after over 30 minutes without errors, with a computed duration of 51 min.

GrammarlyGenerative AI assistance is working with interruptions

Grammarly’s generative AI assistance experienced interruptions in a major-impact incident starting at 2025-02-05 21:05 UTC, lasting 22 min and resolved at 2025-02-05 21:28 UTC. Writing suggestions remained available; no cause or fix was stated.

HHexProjects not loading for some users

Hex reported a critical Main site outage in which projects were not loading for some users, from 2025-02-05 19:03 UTC to 2025-02-05 19:53 UTC (50 min). A fix was implemented by 02-05 19:37 UTC, and the incident was resolved.

SSynthesiaShare Page Videos Not Showing

RRudderStackGoogle Ads offline conversion destination got impacted.

RudderStack reported a major incident affecting Google Ads offline conversion destinations in US and EU regions after Google Ads API v16 deprecation, causing missing or delayed data. It started and resolved at 2025-02-05 13:00 UTC (reported duration: 0 min). Events were replayed to recover data, but customers without backups may still experience data loss.

HHexProjects not loading

Hex projects were not loading on the Main site in a major-impact outage starting 2025-02-04 22:52 UTC and lasting 37 min. A fix was implemented by 2025-02-04 22:56 UTC, and the incident was resolved at 2025-02-04 23:30 UTC.

SubstackSubstack home page and app home tab errors

Substack reported a major incident affecting its website home page and app home tab, with increased errors and Notes missing from some users’ feeds. It lasted 1 h 24 min, from 2025-02-03 19:25 UTC to 2025-02-03 20:49 UTC; Substack restored feeds for all users and reported that error rates had subsided.

PPendoPendo Feedback: Partial Front End Outage

Pendo Feedback experienced a partial front-end outage affecting Feedback - Feedback UI and Feedback UI, rated major by Pendo. It started at 2025-02-03 15:10 UTC and was resolved at 2025-02-03 15:30 UTC, lasting 20 min; no cause or fix was stated.

FFireboltFirebolt V2 UI Experiencing Outage

Firebolt V2 UI access was down in a major incident starting 2025-01-31 00:23 UTC, resolved at 2025-01-31 01:29 UTC; the recorded duration was 1 h 5 min. Programmatic access could also be affected if a user token expired. Firebolt reported full operation restored at 01:15 UTC; no cause was stated.

HHexWeb app outage

Hex reported a critical web app outage affecting the Main site, starting 2025-01-30 17:10 UTC and resolved at 2025-01-30 17:56 UTC (computed duration: 45 min). A fix was implemented by 2025-01-30 17:21 UTC, followed by monitoring; no cause was stated.

HHexProjects not loading

Hex reported a critical Main site outage that prevented projects from loading, starting at 2025-01-30 14:35 UTC and resolved at 2025-01-30 15:55 UTC, lasting 1 h 20 min. Hex reported implementing a fix and monitoring results; no cause was stated.

CCensusIP Address Change for Database Requests

Census reported a critical Sync Engine outage starting 2025-01-30 02:59 UTC: a networking misconfiguration sent database requests from unadvertised IPs, causing syncs to fail for customers whose databases rejected those IPs. Census rolled back to a previous configuration and resolved the incident at 2025-01-30 03:31 UTC; the reported duration was 31 min.

CCriblCribl Cloud Emergency Worker Maintenance

UpstashDisk failure in some Regional Databases

Upstash reported a major outage affecting database reachability in AWS US-EAST-1 and Redis Regional - AWS US-EAST-1. Recorded start and resolution: 2025-01-29 18:03 UTC (computed duration: 0 min); the update reports disk failure at 16:52 UTC and recovery at 17:13 UTC after restarting affected components, with no data lost.

SSpeechmaticsUsers are experiencing issues signing into portal.speechmatics.com.

CodaPartial outage across API request handling and doc processing

Coda reported a major partial outage degrading doc writes through the API server and doc processing, starting 2025-01-29 04:24 UTC. It lasted 1 h 26 min and was resolved at 2025-01-29 05:50 UTC; no cause or specific fix was stated.

OOrbElevated error rates on subscription related endpoints

UUnbounceApp.unbounce.com Currently Not Available

Unbounce reported a major outage of app.unbounce.com starting 2025-01-28 14:01 UTC, with visitors potentially seeing errors when accessing the app. A fix was implemented, and the app was reported up again at 14:30 UTC; the incident was resolved at 2025-01-28 15:10 UTC, with a recorded duration of 1 h 8 min.

OOmni AnalyticsIntermittent Errors in US Environment

Omni Analytics reported a major incident affecting EastUsa: intermittent US-region network errors caused error:500 "Something went wrong" messages; refreshing could help. It started at 2025-01-27 18:13 UTC and was resolved at 2025-01-28 01:11 UTC, lasting 6 h 57 min. The vendor identified the source but did not specify the cause or fix.

CCensusMany syncs failing for 30 minute period

Census reported a major Sync Engine incident starting at 2025-01-25 02:11 UTC, resolved after 16 min. The vendor said a misconfiguration caused many syncs to fail approximately 2025-01-25 01:08 to 01:38 UTC; it deployed a fix and reported no further failures.

PPipedreamGateway timeouts in HTTP workflow executions

Pipedream reported a major incident causing gateway timeouts in HTTP workflow executions, affecting Backend - Workflows - HTTP and Workflows - HTTP. It started at 2025-01-24 15:30 UTC and was resolved at 2025-01-24 17:37 UTC, lasting 2 h 7 min; no cause or specific fix was stated.

GrammarlyWe're experiencing a service interruption

Grammarly reported a critical service interruption affecting Grammarly Editor, with some users unable to log in or create accounts and slower loading. The incident ran from 2025-01-23 14:41 UTC to 2025-01-23 15:48 UTC, lasting 1 h 7 min. Grammarly implemented a fix and confirmed service was back to normal.

SSambaNovaRegional Login to SambaNova Cloud - Japan recieving errors

WarpRequests to warp server failing

GrammarlyGrammarly is experiencing a service interruption, some Grammarly users may be unable to log in

Grammarly reported a critical service interruption that left some users unable to log in, starting at 2025-01-22 19:35 UTC and resolved at 2025-01-22 21:58 UTC, lasting 2 h 23 min. The vendor identified the root cause without disclosing it and implemented a fix; affected components included Login, Writing Suggestions, Generative AI assistance, and Billing Service.

FFireboltUS-EAST-1 Engine Operations

Firebolt reported a major US-EAST-1 outage from 2025-01-22 19:22 UTC to 2025-01-22 20:58 UTC, lasting 1 h 36 min. Customers could not start engines or use USE ENGINE and could encounter Develop space access and engine/account listing API issues; queries on running engines were expected to work. Firebolt deployed a fix and confirmed full restoration.

SSynthesiaVideo Generation Issues

DropboxDocSend unavailable

DocSend was unavailable in a major-impact incident that started at 2025-01-22 15:45 UTC and was resolved at 2025-01-22 15:52 UTC; the computed duration was 6 min. Customers could have experienced errors or long load times. Dropbox did not state a cause or fix.

PrismaAccelerate issue provisioning new query engines in us-east-1

SubstackIncreased latency and errors

Substack reported a major incident causing increased latency and errors across its websites and applications, including custom domains, publications and writer dashboards. It started at 2025-01-21 23:44 UTC and was resolved at 2025-01-22 00:36 UTC, lasting 52 min; no specific cause or fix was stated.

AsanaPartial Outage

Asana reported a major partial outage affecting API, App, Automations and Background Actions, Mobile, and Webhooks and Event Streams, including US components. It started at 2025-01-17 21:50 UTC and was resolved at 2025-01-17 23:39 UTC (1 h 48 min); a fix was implemented at 01-17 22:05 UTC.

HHexHTML and styled dataframes not rendering

Hex reported a major Main site incident in which HTML and styled dataframes were not rendering. It started at 2025-01-17 15:12 UTC and was resolved at 2025-01-17 16:34 UTC, lasting 1 h 22 min; no cause or specific fix was stated.

QuickBooks OnlineQBO Create invoice forced logged out issue

QuickBooks Online reported a major incident in Canada and the United States that forced some customers out of their accounts when creating transactions or accessing the Sales Tax Center. It started at 2025-01-16 20:52 UTC and was resolved at 2025-01-17 15:08 UTC, lasting 18 h 16 min; an earlier update at 20:59 UTC had already reported service restored.

GrammarlySome Grammarly accounts are temporarily deactivated

Some Grammarly accounts were temporarily deactivated, affecting Login and prompting deactivation emails. Grammarly rated the incident major; it ran from 2025-01-16 01:57 UTC to 2025-01-16 03:54 UTC, lasting 1 h 57 min. All affected accounts were reactivated; Grammarly said it identified the cause but did not disclose it.

Jupyter (Anaconda)anaconda.org elevated error rates

Anaconda.org experienced elevated error rates affecting its front-end web service, channels and repository services, with major impact. The incident ran from 2025-01-15 13:08 UTC to 2025-01-15 14:35 UTC, lasting 1 h 26 min. Anaconda reported implementing a fix at 14:05 UTC and confirmed successful resolution after monitoring.

CodecovUnable to login to Codecov

Codecov reported a major outage preventing login using any service, affecting Codecov Backend and Codecov Frontend; uploads appeared unaffected. It started at 2025-01-15 00:45 UTC and was resolved at 2025-01-15 14:03 UTC, with a reported duration of 13 h 17 min. A fix was implemented by 01-15 13:50 UTC; no cause was stated.

JFrogJFrog services intermittent degradation affecting a subset of customers in AWS US West 2 region

JFrog reported major intermittent service disruptions affecting a subset of customers in AWS US West 2 (Oregon), starting 2025-01-13 18:34 UTC and resolved 2025-01-13 20:15 UTC, lasting 1 h 40 min. Affected components included Artifactory, Distribution, Pipelines, security services, Webhook and Web UI; JFrog reported implementing a fix at 19:52 UTC.

CodaAttachment and image processing delays

Coda reported a major outage preventing virus scanning of attachments and images, delaying their processing and availability to document collaborators; uploads were stored successfully. It started at 2025-01-10 04:31 UTC and was resolved at 2025-01-10 15:25 UTC, lasting 10 h 53 min.

TTeleportNew Tenant Onboarding Failures

MapboxTraffic Maps V1 Degraded Performance

Mapbox reported major degraded performance affecting the mapbox.mapbox-traffic-v1 tileset and Traffic Congestion Visual Layer from 2025-01-09 18:37 UTC to 2025-01-10 02:16 UTC, lasting 7 h 39 min. Mapbox identified the root cause without detailing it, implemented a fix, and confirmed resolution.