Companies/Amazon Web Services/Incidents

Amazon Web Services: Increased Error Rates and Latencies (us-east-1)

flowchart LR
  root["AWS us-east-1<br/>posted 07:11 UTC"]:::root
  before["Posted before AWS did<br/>5 vendors, 4 blamed AWS<br/>Salesforce, Descope, Ashby +2"]:::blames
  root --> before
  h1["Within an hour<br/>18 vendors, 17 blamed AWS<br/>Asana, Firebolt, Monday.com +15"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>30 vendors, 29 blamed AWS<br/>SingleStore, Loops, Buildkite +27"]:::blames
  root --> h4
  later["Later<br/>45 vendors, 44 blamed AWS<br/>RudderStack, Airbyte, Unbounce +42"]:::blames
  root --> later
  p0["Twilio<br/>posted +15 min"]:::blames
  root --> p0
  k0["BigCommerce<br/>+2 h 46 min"]:::hop2
  p0 -->|"blamed Twilio"| k0
  p1["Webflow<br/>posted +4 h 37 min, did not name AWS"]:::timing
  root -.-> p1
  k1["Rootly<br/>+10 h 23 min"]:::hop2
  p1 -->|"blamed Webflow"| k1
  p2["Baseten<br/>posted +4 min"]:::blames
  root --> p2
  k2["Zed<br/>+26 h 20 min"]:::hop2
  p2 -->|"blamed Baseten"| k2
  class root me

Affected components: AWS Account ManagementAWS Private Certificate AuthorityAmazon Managed Workflows for Apache AirflowAWS AmplifyAmazon API GatewayAWS AppConfigAmazon AppFlowAmazon CloudWatch Application InsightsAmazon WorkSpaces ApplicationsAWS AppSyncAWS Launch WizardAmazon Managed Service for PrometheusAmazon AthenaAWS IoT GreengrassAWS IoT Device ManagementAWS IoT CoreAWS Partner CentralAWS WAFAWS B2B Data InterchangeAWS BatchAmazon BedrockAWS Billing ConsoleAmazon ChimeAWS Client VPNAWS CloudFormationAmazon CloudFrontAWS CloudHSMAWS CloudTrailAWS Cloud WANAmazon CloudWatchAWS CodeBuildAmazon CognitoAmazon ComprehendAWS ConfigAmazon ConnectAWS Control TowerAWS DataSyncAmazon DataZoneAWS Deadline CloudAWS Direct ConnectAWS Directory ServiceAWS Database Migration ServiceAmazon DocumentDBAWS Elastic Disaster RecoveryAmazon Aurora DSQL ServiceAmazon DynamoDBAWS Systems ManagerAmazon Elastic Compute CloudAmazon Elastic Container RegistryAmazon Elastic Container ServiceAmazon Elastic Kubernetes ServiceAmazon ElastiCacheAWS Elastic BeanstalkAmazon Elastic File SystemAmazon OpenSearch ServiceAmazon Elastic Load BalancingAWS ElementalAmazon EMR ServerlessAmazon Elastic MapReduceAWS End User MessagingAmazon EventBridgeAWS Elastic VMWare ServiceAmazon Kinesis FirehoseAWS Firewall ManagerAmazon FSxAmazon GameLift StreamsAmazon GameLift ServersAWS Global AcceleratorAWS GlueAmazon Managed GrafanaAmazon GuardDutyAWS HealthImagingAWS HealthLakeAWS Identity and Access ManagementAWS IAM Identity CenterEC2 Image BuilderAWS IoT AnalyticsAWS IoT EventsAWS IoT FleetWiseAWS IoT SiteWiseAmazon VPC IP Address ManagerAmazon Interactive Video ServiceAmazon Managed Streaming for Apache KafkaAmazon KendraAmazon Managed Service for Apache FlinkAmazon Kinesis Data StreamsAmazon Kinesis Video StreamsAWS Lake FormationAWS LambdaAWS License ManagerAmazon Location ServiceAmazon EC2 Instance ConnectAWS Application Migration ServiceAmazon MQAWS NAT GatewayAmazon NeptuneAWS Network FirewallAWS HealthOmicsAWS OrganizationsAWS OutpostsAWS Payment CryptographyAWS Parallel Computing ServiceAmazon PinpointAmazon PollyAWS VPCE PrivateLinkAmazon Q BusinessAmazon Quick SuiteAmazon Relational Database ServiceAmazon RedshiftAmazon RekognitionAWS Resource GroupsAmazon Simple Storage ServiceAmazon SageMakerAmazon EventBridge SchedulerAWS Secrets ManagerAWS Security Incident ResponseAmazon Security LakeAmazon Simple Email ServiceAmazon Simple Notification ServiceAmazon Simple Queue ServiceAWS Systems Manager for SAPAWS Step FunctionsAWS Storage GatewayAWS Security Token ServiceAWS Support APIAWS Support CenterAmazon Simple Workflow ServiceAmazon TextractAmazon WorkSpaces Thin ClientAmazon TimestreamTraffic MirroringAmazon TranscribeAWS Transfer FamilyAWS TransformAWS Transit GatewayAmazon TranslateAWS Verified AccessAmazon VPC LatticeAWS Site-to-Site VPNAWS WickrAmazon WorkMailAmazon WorkSpaces

Timeline of updates

1. update2025-10-20 07:11 UTC

   We are investigating increased error rates and latencies for multiple AWS services in the US-EAST-1 Region. We will provide another update in the next 30-45 minutes.

2. update2025-10-20 07:51 UTC

   We can confirm increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause. We will provide an update in 45 minutes, or sooner if we have additional information to share.

3. update2025-10-20 08:26 UTC

   We can confirm significant error rates for requests made to the DynamoDB endpoint in the US-EAST-1 Region. This issue also affects other AWS Services in the US-EAST-1 Region as well. During this time, customers may be unable to create or update Support Cases. Engineers were immediately engaged and are actively working on both mitigating the issue, and fully understanding the root cause. We will continue to provide updates as we have more information to share, or by 2:00 AM.

4. update2025-10-20 09:01 UTC

   We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery. This issue also affects other AWS Services in the US-EAST-1 Region. Global services or features that rely on US-EAST-1 endpoints such as IAM updates and DynamoDB Global tables may also be experiencing issues. During this time, customers may be unable to create or update Support Cases. We recommend customers continue to retry any failed requests. We will continue to provide updates as we have more information to share, or by 2:45 AM.

5. update2025-10-20 09:22 UTC

   We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

6. update2025-10-20 09:27 UTC

   We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information.

7. update2025-10-20 10:03 UTC

   We continue to observe recovery across most of the affected AWS Services. We can confirm global services and features that rely on US-EAST-1 have also recovered. We continue to work towards full resolution and will provide updates as we have more information to share.

8. update2025-10-20 10:35 UTC

   The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2 instances (or services that launch EC2 instances such as ECS) in the US-EAST-1 Region are still experiencing increased error rates. We continue to work toward full resolution. If you are still experiencing an issue resolving the DynamoDB service endpoints in US-EAST-1, we recommend flushing your DNS caches. We will provide an update by 4:15 AM, or sooner if we have additional information to share.

9. update2025-10-20 11:08 UTC

   We are continuing to work towards full recovery for EC2 launch errors, which may manifest as an Insufficient Capacity Error. Additionally, we continue to work toward mitigation for elevated polling delays for Lambda, specifically for Lambda Event Source Mappings for SQS. We will provide an update by 5:00 AM PDT.

10. update2025-10-20 11:48 UTC

    We continue to work to fully restore new EC2 launches in US-EAST-1. We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically. We are pursuing further mitigation steps to recover Lambda’s polling delays for Event Source Mappings for SQS. AWS features that depend on Lambda’s SQS polling capabilities such as Organization policy updates are also experiencing elevated processing times. We will provide an update by 5:30 AM PDT.

11. update2025-10-20 12:10 UTC

    We confirm that we have now recovered processing of SQS queues via Lambda Event Source Mappings. We are now working through processing the backlog of SQS messages in Lambda queues.

12. update2025-10-20 12:48 UTC

    We are making progress on resolving the issue with new EC2 instance launches in the US-EAST-1 Region and are now able to successfully launch new instances in some Availability Zones. We are applying similar mitigations to the remaining impacted Availability Zones to restore new instance launches. As we continue to make progress, customers will see an increasing number of successful new EC2 launches. We continue to recommend that customers launch new EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. We also wanted to share that we are continuing to successfully process the backlog of events for both EventBridge and Cloudtrail. New events published to these services are being delivered normally and are not experiencing elevated delivery latencies. We will provide an update by 6:30 AM PDT or sooner if we have additional information to share.

13. update2025-10-20 13:42 UTC

    We have applied multiple mitigations across multiple Availability Zones (AZs) in US-EAST-1 and are still experiencing elevated errors for new EC2 instance launches. We are rate limiting new instance launches to aid recovery. We will provide an update at 7:30 AM PDT or sooner if we have additional information.

14. update2025-10-20 14:14 UTC

    We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information.

15. update2025-10-20 14:29 UTC

    We have confirmed multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for the connectivity issues and are continuing to investigate the root cause.

16. update2025-10-20 15:04 UTC

    We continue to investigate the root cause for the network connectivity issues that are impacting AWS services such as DynamoDB, SQS, and Amazon Connect in the US-EAST-1 Region. We have identified that the issue originated from within the EC2 internal network. We continue to investigate and identify mitigations.

17. update2025-10-20 15:43 UTC

    We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations.

18. update2025-10-20 16:13 UTC

    We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.

19. update2025-10-20 17:03 UTC

    We continue to apply mitigation steps for network load balancer health and recovering connectivity for most AWS services. Lambda is experiencing function invocation errors because an internal subsystem was impacted by the network load balancer health checks. We are taking steps to recover this internal Lambda system. For EC2 launch instance failures, we are in the process of validating a fix and will deploy to the first AZ as soon as we have confidence we can do so safely. We will provide an update by 10:45 AM PDT.

20. update2025-10-20 17:38 UTC

    Our mitigations to resolve launch failures for new EC2 instances are progressing and the internal subsystems of EC2 are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. We are applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside. We will provide an update by 11:30 AM PDT.

21. update2025-10-20 18:22 UTC

    Our mitigations to resolve launch failures for new EC2 instances continue to progress and we are seeing increased launches of new EC2 instances and decreasing networking connectivity issues in the US-EAST-1 Region. We are also experiencing significant improvements to Lambda invocation errors, especially when creating new execution environments (including for Lambda@Edge invocations). We will provide an update by 12:00 PM PDT.

22. update2025-10-20 19:15 UTC

    We continue to observe recovery across all AWS services, and instance launches are succeeding across multiple Availability Zones in the US-EAST-1 Regions. For Lambda, customers may face intermittent function errors for functions making network requests to other services or systems as we work to address residual network connectivity issues. To recover Lambda’s invocation errors, we slowed down the rate of SQS polling via Lambda Event Source Mappings. We are now increasing the rate of SQS polling as we experience more successful invocations and reduced function errors. We will provide another update by 1:00 PM PDT.

23. update2025-10-20 20:03 UTC

    Service recovery across all AWS services continues to improve. We continue to reduce throttles for new EC2 Instance launches in the US-EAST-1 Region that were put in place to help mitigate impact. Lambda invocation errors have fully recovered and function errors continue to improve. We have scaled up the rate of polling SQS queues via Lambda Event Source Mappings to pre-event levels. We will provide another update by 1:45 PM PDT.

24. update2025-10-20 20:52 UTC

    We have continued to reduce throttles for EC2 instance launches in the US-EAST-1 Region and we continue to make progress toward pre-event levels in all Availability Zones (AZs). AWS services such as ECS and Glue, which rely on EC2 instance launches will recover as the successful instance launch rate improves. We see full recovery for Lambda invocations and are working through the backlog of queued events which we expect to be full processed in approximately in the next two hours. We will provide another update by 2:30 PM PDT.

25. update2025-10-20 21:48 UTC

    We have restored EC2 instance launch throttles to pre-event levels and EC2 launch failures have recovered across all Availability Zones in the US-EAST-1 Regions. AWS services which rely on EC2 instance launches such as Redshift are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours. We can confirm that Connect is handling new voice and chat sessions normally. There is a backlog of analytics and reporting data that we must process and anticipate that we will have worked through the backlog over the next two hours. We will provide an update by 3:30 PM PDT.

26. resolved2025-10-20 22:53 UTC

    Between 11:49 PM PDT on October 19 and 2:24 AM PDT on October 20, we experienced increased error rates and latencies for AWS Services in the US-EAST-1 Region. Additionally, services or features that rely on US-EAST-1 endpoints such as IAM and DynamoDB Global Tables also experienced issues during this time. At 12:26 AM on October 20, we identified the trigger of the event as DNS resolution issues for the regional DynamoDB service endpoints. After resolving the DynamoDB DNS issue at 2:24 AM, services began recovering but we had a subsequent impairment in the internal subsystem of EC2 that is responsible for launching EC2 instances due to its dependency on DynamoDB. As we continued to work through EC2 instance launch impairments, Network Load Balancer health checks also became impaired, resulting in network connectivity issues in multiple services such as Lambda, DynamoDB, and CloudWatch. We recovered the Network Load Balancer health checks at 9:38 AM. As part of the recovery effort, we temporarily throttled some operations such as EC2 instance launches, processing of SQS queues via Lambda Event Source Mappings, and asynchronous Lambda invocations. Over time we reduced throttling of operations and worked in parallel to resolve network connectivity issues until the services fully recovered. By 3:01 PM, all AWS services returned to normal operations. Some services such as AWS Config, Redshift, and Connect continue to have a backlog of messages that they will finish processing over the next few hours. Additional details regarding this event are available <a href="https://aws.amazon.com/message/101925/">here.</a>

Changes Rewindex caught on this incident

• Oct 20, 2025Critical incident: Increased Error Rates and Latencies (us-east-1)

Impact, status and times are the vendor’s own, as last archived. View on health.aws.amazon.com (opens in new tab).

Amazon Web Services profileRecent Amazon Web Services incidentsMarkdownJSON