Companies/Amplitude/Incidents

Amplitude: Data collection loss from JS SDK file hosting bug

Timeline of updates

1. resolved2019-10-18 23:30 UTC

   On Friday we broke how our JS SDK files were being hosted which caused data loss. We fixed the issue on Monday. Who was affected? Customers using our JS SDK via the snippet (https://developers.amplitude.com/#installation (opens in new tab)). Customers who use npm to include our SDK won't have been affected. What happened? We had intended to increase the HTTP header Cache-Control max-age on all SDK files. But we accidentally dropped and overwrote other important headers (Content-Type and Content-Encoding) in a way that broke how browsers loaded them. We fixed the issue by re-setting the correct headers. When did this happen? We broke the SDK file headers at approx 16:45 PDT (23:45 UTC) 2019-10-18. We fixed the headers at approx 09:30 PDT (16:30 UTC) 2019-10-21. What are we doing to prevent this ever happening again? At a low level the issue stemmed from us not understanding how boto2's set_remote_metadata method worked (we use AWS CloudFront and S3 for our CDN). We at least know not to use this API in the future. We'll also spend some time thinking about how we can tighten up controls around the SDK files to prevent similar accidental changes in the future.

From Amplitude’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on status.amplitude.com (opens in new tab).

Amplitude profileRecent Amplitude incidentsAll outages on October 18, 2019Amplitude in October 2019MarkdownJSON