Changes / Mixpanel / Mixpanel.com and APIs experiencing elevated errors
Mixpanel: Critical incident: Mixpanel.com and APIs experiencing elevated errors
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Mixpanel.com and APIs experiencing elevated errors
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2025-06-12 18:29 UTC
Before: not present
After: 2025-06-13 02:04 UTC
Before: not present
After: Application Availability - Application Availability (EU), Application Availability - Application Availability (IN), Application Availability - Application Availability (US), Application Availability (EU), Application Availability (IN), Application Availability (US), Data Export, Data Ingestion - Ingestion API Availability (EU), Data Ingestion - Ingestion API Availability (IN), Data Ingestion - Ingestion API Availability (US), Data Ingestion - Warehouse Connectors, Ingestion API Availability (EU), Ingestion API Availability (IN), Ingestion API Availability (US), Warehouse Connectors
Before: not present
After: Mixpanel is experiencing a website outage due to an issue with Google Cloud Platform. Data ingestion and export are affected as well. We are working to restore service as quickly as possible.
Before: not present
After: Due to the Google Cloud Platform outage, Mixpanel's Ingestion API endpoints, including event data (via /track and /import), user profile data (via /engage), and group profile data (via /groups) are experiencing elevated error rates. Data sent to these APIs that resulted in an error will not be stored on our end and should be re-sent.
<strong>Potential Action Required</strong>
If you are using one of our client-side SDKs which has built-in retry logic OR if you are using a server-side SDK and have already implemented <a href="https://developer.mixpanel.com/reference/import-events?#rate-limits" target="_blank">the recommended retry behavior</a>, your requests to our ingestion APIs will likely be ingested once the services are restored and no additional action is necessary.
If you are using a server-side SDK and have not implemented the recommended retry behavior, you will need to resend data for the affected period. The data you are resending may include events already seen in Mixpanel, so please ensure those events maintain <strong>the same $insert_id, time, distinct_id, and event name</strong>, so that Mixpanel properly deduplicates the data. If you have not already implemented automatic retry behavior for your ingestion pipeline we would highly recommend doing so.
We are committed to delivering accurate, timely analysis to support your critical decision-making, and we prioritize fast, high-quality customer support. If you have any questions or concerns, please feel free to reach out.
Before: not present
After: Google has reported that their infrastructure is almost fully recovered. Mixpanel's EU and IN servers are fully recovered, while the US servers are still recovering. The website is back up, and error rates on the Ingestion API endpoints are decreasing. We are starting to receive a large amount of data as services recovers, so ingestion will be delayed as it processes through our system.
We really appreciate your patience as we work to restore everything back to normal. If you have any questions or concerns, please feel free to reach out.
Before: not present
After: Data ingestion in EU and IN servers have caught up and is no longer delayed. Data ingestion in US servers is still catching up.
Before: not present
After: The Mixpanel website and Export API have recovered. Data ingestion in US servers is still catching up.
Before: not present
After: Data ingestion in US servers has caught up and is no longer delayed. We will continue to monitor the situation.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Mixpanel.com and APIs experiencing elevated errors (this change) | Major incident |