Rewindex

ChangesAmazon Web ServicesIncreased connectivity issues and API Error Rates (me-south-1)

Amazon Web Services: Critical incident: Increased connectivity issues and API Error Rates (me-south-1)

Major incidentcritical (vendor)Changed Detected health.aws.amazon.com (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what Amazon Web Services removed; blue is what they added. Muted text is unchanged context.

AddedTitle#1
Before: not present
After: Increased connectivity issues and API Error Rates (me-south-1)
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: investigating
AddedStart time#4
Before: not present
After: 2026-03-02 05:56 UTC
AddedAffected components#5
Before: not present
After: Amazon API Gateway, Amazon Athena, Amazon Bedrock, Amazon CloudFront, Amazon CloudWatch, Amazon CloudWatch Application Insights, Amazon CloudWatch Internet Monitor, Amazon CloudWatch RUM, Amazon CloudWatch Synthetics, Amazon Cognito, Amazon Data Firehose, Amazon Data Lifecycle Manager, Amazon Detective, Amazon DynamoDB, Amazon EC2 Instance Connect, Amazon ElastiCache, Amazon Elastic Compute Cloud, Amazon Elastic Container Registry, Amazon Elastic Container Service, Amazon Elastic File System, Amazon Elastic Kubernetes Service, Amazon Elastic Load Balancing, Amazon Elastic MapReduce, Amazon EMR Serverless, Amazon EventBridge, Amazon EventBridge Scheduler, Amazon FreeRTOS, Amazon FSx, Amazon GameLift Streams, Amazon Glacier, Amazon GuardDuty, Amazon Inspector, Amazon Keyspaces, Amazon Kinesis Data Streams, Amazon Kinesis Video Streams, Amazon Macie, Amazon Managed Service for Apache Flink, Amazon Managed Streaming for Apache Kafka, Amazon Managed Workflows for Apache Airflow, Amazon MQ, Amazon Neptune, Amazon OpenSearch Service, Amazon Polly, Amazon Redshift, Amazon Relational Database Service, Amazon Route 53, Amazon Route 53 Private DNS, Amazon Route 53 Resolver, Amazon S3 Replication Time Control, Amazon SageMaker, Amazon Simple Email Service, Amazon Simple Notification Service, Amazon Simple Queue Service, Amazon Simple Storage Service, Amazon Simple Workflow Service, Amazon Transcribe, Amazon Verified Permissions, Amazon Virtual Private Cloud, Amazon VPC IP Address Manager, Amazon VPC Lattice, Auto Scaling, AWS Amplify, AWS Amplify Admin, AWS AppConfig, AWS Application Migration Service, AWS App Mesh, AWS AppSync, AWS Backup, AWS Batch, AWS Certificate Manager, AWS Client VPN, AWS Cloud9, AWS CloudFormation, AWS CloudHSM, AWS Cloud Map, AWS CloudShell, AWS CloudTrail, AWS Cloud WAN, AWS CodeBuild, AWS CodeCommit, AWS CodeDeploy, AWS CodePipeline, AWS Compute Optimizer, AWS Config, AWS Console Mobile App, AWS Control Tower, AWS Database Migration Service, AWS DataSync, AWS Direct Connect, AWS Directory Service, AWS Elastic Beanstalk, AWS Elastic Disaster Recovery, AWS End User Messaging, AWS Fargate, AWS Fault Injection Service, AWS Firewall Manager, AWS Global Accelerator, AWS Glue, AWS Ground Station, AWS IAM Identity Center, AWS Identity and Access Management Roles Anywhere, AWS Internet Connectivity, AWS IoT Core, AWS IoT Device Defender, AWS IoT Device Management, AWS Key Management Service, AWS Lake Formation, AWS Lambda, AWS Launch Wizard, AWS License Manager, AWS Management Console, AWS NAT Gateway, AWS Network Firewall, AWS Outposts, AWS Private CA Connector for Active Directory, AWS Private CA Connector for SCEP, AWS Private Certificate Authority, AWS Resilience Hub, AWS Resource Access Manager, AWS Resource Explorer, AWS Resource Groups, AWS Resource Groups Tagging API, AWS Secrets Manager, AWS Security Hub, AWS Security Incident Response, AWS Security Token Service, AWS Serverless Application Repository, AWS Service Catalog, AWS Service Quotas, AWS Sign-In, AWS Site-to-Site VPN, AWS Step Functions, AWS Storage Gateway, AWS Systems Manager, AWS Systems Manager for SAP, AWS Transfer Family, AWS Transit Gateway, AWS VPCE PrivateLink, AWS WAF, AWS Well-Architected Tool, AWS X-Ray, Control Catalog, EC2 Image Builder, EC2 Serial Console, Infrastructure Performance, Network Access Analyzer, Reachability Analyzer, Traffic Mirroring, VM Import/Export
AddedUpdateupdate#6
Before: not present
After: We are investigating increased API error rates in a single Availability Zone (mes1-az2) in the ME-SOUTH-1 Region.
AddedUpdateupdate#7
Before: not present
After: We are investigating connectivity and power issues affecting APIs and instances in a single Availability Zone (mes1-az2) in the ME-SOUTH-1 Region due to a localized power issue. Existing instances in this zone will also be affected. Other AWS Services may also be experiencing increased errors and latencies for their workflows, and we are working to route requests away from this affected Availability Zone. We recommend customers make use of other Availability Zones at this time. During this time, we are also experiencing delays in propagating DNS changes for Route53 to pops (Points of Presence) in ME-SOUTH-1.  Targeting new launches using RunInstances in the remaining AZs should succeed. Existing instances in the other AZs are not affected.
AddedUpdateupdate#8
Before: not present
After: We continue to work on a localized power issue affecting a single Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. In the impacted Availability Zone, EC2 Instances, DB Instances, EBS Volumes, and other AWS Services are also experiencing elevated error rates and latencies for some workflows. As part of our recovery effort, we have shifted traffic away from the impacted Availability Zone for most services. We recommend customers utilize one of the other Availability Zones in the ME-SOUTH-1 Region, as existing instances in other AZs remain unaffected by this issue. We are actively working to restore power and connectivity, at which time we will begin recovering affected resources. Currently, we expect recovery to take many hours. We will provide an update by 2:30 AM PST, or sooner if we have additional information to share.
AddedUpdateupdate#9
Before: not present
After: We continue to work toward restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. At this time, some AWS services have shifted traffic away from the affected Availability Zone and are seeing recovery for their affected operations and workflows. EC2 Instances, EBS Volumes, and other resources impacted in the affected Availability Zone will require a longer recovery timeline. Power has not yet been restored to the affected Availability Zone. If immediate recovery is required, we recommend customers restore from EBS Snapshots and/or launch replacement resources in one of the unaffected Availability Zones or an alternate Region. In parallel, we are actively working on reducing the error rates and latencies that some customers are experiencing with EC2 APIs. For now, we recommend continuing to retry any failed API requests. We will provide an update by 6:00 AM PST on March 2, or sooner if we have additional information to share.
AddedUpdateupdate#10
Before: not present
After: We continue to work toward restoring power in the impacted Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. Meanwhile, EC2 instance and networking APIs have been restored for the other Availability Zones. Additionally, we have made improvements to the availability of RDS multi-AZ databases while operating with the impaired Availability Zone. These improvements will help customers create database exports to preserve data, and we recommend customers with databases in the affected Availability Zone consider creating exports as a precautionary measure. EC2 Instances, EBS Volumes, and other resources impacted in the affected Availability Zone will require a longer recovery timeline, as power has not yet been restored. We are expecting recovery to take at least a day, as it requires repair of facilities, cooling and power systems, coordination with local authorities, and careful assessment to ensure the safety of our operators. If immediate recovery is required, we recommend customers restore from EBS Snapshots and/or launch replacement resources in one of the unaffected Availability Zones or an alternate AWS Region. We will provide an update by 11:00 AM PST on March 2, or sooner if we have additional information to share.
AddedUpdateupdate#11
Before: not present
After: We continue to work towards restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. We currently expect our recovery efforts to take at least a day. Our current guidance regarding immediate recovery remains unchanged from our previous update. Customers are able to disassociate Elastic IP addresses from resources in the affected Availability Zone and associate those with resources in the unaffected Availability Zones. This can be done by specifying --allow-reassociation when attempting to associate the Elastic IP to the new resource. We will provide you with further updates by 2:00 PM PST or sooner if new information becomes available.
AddedUpdateupdate#12
Before: not present
After: We continue to work towards restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. We have no updated guidance on expected recovery times, and still expect this to take at least a day to fully restore power and connectivity. We continue to advise customers to launch replacement resources in one of the unaffected Availability Zones or an alternate AWS Region. At this time we recommend that customers that are capable of backing up data outside of the region consider doing so. You can view the current status of affected AWS services below. We will provide you with another update by 7:00 PM PST, or sooner if we have additional information to share.
AddedUpdateupdate#13
Before: not present
After: We are providing an update on the ongoing service disruptions affecting the AWS Middle East (UAE) Region (ME-CENTRAL-1) and the AWS Middle East (Bahrain) Region (ME-SOUTH-1). Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes. In the UAE, two of our facilities were directly struck, while in Bahrain, a drone strike in close proximity to one of our facilities caused physical impacts to our infrastructure. These strikes have caused structural damage, disrupted power delivery to our infrastructure, and in some cases required fire suppression activities that resulted in additional water damage. We are working closely with local authorities and prioritizing the safety of our personnel throughout our recovery efforts. In the ME-CENTRAL-1 (UAE) Region, two of our three Availability Zones (mec1-az2 and mec1-az3) remain significantly impaired. The third Availability Zone (mec1-az1) continues to operate normally, though some services have experienced indirect impact due to dependencies on the affected zones. In the ME-SOUTH-1 (Bahrain) Region, one facility has been impacted. Across both regions, customers are experiencing elevated error rates and degraded availability for services including Amazon EC2, Amazon S3, Amazon DynamoDB, AWS Lambda, Amazon Kinesis, Amazon CloudWatch, Amazon RDS, and the AWS Management Console and CLI. We are working to restore full service availability as quickly as possible, though we expect recovery to be prolonged given the nature of the physical damage involved. In parallel with efforts to restore the physical infrastructure at the affected sites, we are pursuing multiple software-based recovery paths that do not depend on the underlying facilities being fully brought back online. For Amazon S3 and Amazon DynamoDB, we are actively working to restore data access and service availability through software mitigations, including deploying updates to enable S3 to operate within the current infrastructure constraints and remediating impaired DynamoDB tables to restore read and write availability for dependent services. Our focus on restoring these foundational services is deliberate, as recovery of Amazon S3 and Amazon DynamoDB will in turn enable a broad range of dependent AWS services to recover. For other affected service APIs, we are deploying targeted software updates to reduce error rates and restore functionality where possible, independent of the physical recovery timeline. We are also working to restore access to the AWS Management Console and CLI through network-level changes that route traffic away from the affected infrastructure. While these software-based mitigations can address many of the service-level impacts, some recovery actions are constrained by the physical state of the affected facilities — meaning that full restoration of certain services will require the underlying infrastructure to be repaired and brought back online. Across all services, our teams are working in parallel on both the physical restoration of the affected facilities and these software-based mitigations, with the goal of restoring as much customer access as possible as quickly as possible, even ahead of full infrastructure recovery. In addition, we are prioritizing the restoration of services and tools that enable customers to back up and migrate their data and applications out of the affected regions. Finally, even as we work to restore these facilities, the ongoing conflict in the region means that the broader operating environment in the Middle East remains unpredictable. We recommend that customers with workloads running in the Middle East consider taking action now to backup data and potentially migrate your workloads to alternate AWS Regions. We recommend customers exercise their disaster recovery plans, recover from remote backups stored in other regions, and update their applications to direct traffic away from the affected regions. For customers requiring guidance on alternate regions, we recommend considering AWS Regions in the United States, Europe, or Asia Pacific, as appropriate for your latency and data residency requirements. We will continue to provide updates as recovery progresses and as the situation evolves. Our next update will be provided by 9:00 PM PST on March 2, 2026, or sooner if new information becomes available.
AddedUpdateupdate#14
Before: not present
After: We continue to work towards restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. We have no updated guidance on expected recovery times, and still expect this to take at least a day to fully restore power and connectivity. AWS infrastructure is designed to be highly resilient, but given the uncertainty of the current situation, we encourage our customers to replicate Amazon S3 and critical data from the ME-SOUTH-1 Region to another AWS Region. For customers requiring guidance on alternate regions, we recommend considering AWS Regions in the United States, Europe, or Asia Pacific, as appropriate for your latency and data residency requirements. We will provide another update by March 3 at 3:00 AM PST, or sooner if new information becomes available. For more information on Cross-Region Replication, refer [1]. For more information on S3 Batch Replication, see [2]. For a simple script to quickly set up and start S3 Replication, see [3]. If you have questions or concerns, please contact AWS Support [4]. [1] <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html">https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html</a> [2] <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html">https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html</a> [3] <a href="https://github.com/awslabs/aws-support-tools/blob/master/S3/Setup_Replication/setup_replication.py">https://github.com/awslabs/aws-support-tools/blob/master/S3/Setup_Replication/setup_replication.py</a> [4] <a href="https://aws.amazon.com/support">https://aws.amazon.com/support</a>
AddedUpdateupdate#15
Before: not present
After: We continue to work toward restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. The overall state of the region remains largely unchanged from our previous update. At this time, we have no updated guidance on expected timelines for fully restoring power and connectivity. We are taking all necessary steps to support the recovery process. While progress is being made, significant work remains before full restoration is complete. Given the ongoing uncertainty, we encourage customers to replicate their Amazon S3 data and other critical data from the ME-SOUTH-1 Region to another AWS Region, using the guidance provided in our previous update. We will continue to provide updates as recovery progresses and as the situation evolves. Our next update will be provided by 6:00 AM PST on March 3, or sooner if new information becomes available.
AddedUpdateupdate#16
Before: not present
After: Recovery efforts in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region are ongoing, with the situation remaining consistent with our last update. We have no change to expected timelines for fully restoring power and connectivity. While progress is being made, significant work remains before full restoration is complete. We continue to recommend customers launch replacement resources in one of the unaffected Availability Zones or an alternate AWS Region. Given the extended nature of this event, we continue to encourage customers to replicate Amazon S3 data and other critical workloads from ME-SOUTH-1 to another AWS Region using the guidance shared previously. We will provide our next update by 12:00 PM PST on March 3, or sooner if conditions change.
AddedUpdateupdate#17
Before: not present
After: We are providing an update on the ongoing service disruptions affecting the AWS Middle East (Bahrain) Region (ME-SOUTH-1). We continue to make progress on recovery efforts across multiple workstreams. With the immediate phase of this event now better understood, we are moving to a more targeted communication model. Going forward, updates will be delivered directly to affected customers through the AWS Personal Health Dashboard. Customers who require assistance with this event are encouraged to contact AWS Support through the AWS Management Console or the AWS Support Center. We continue to strongly recommend that customers with workloads running in the Middle East take action now to migrate those workloads to alternate AWS Regions. Customers should enact their disaster recovery plans, recover from remote backups stored in other Regions, and update their applications to direct traffic away from the affected Regions. For customers requiring guidance on alternate regions, we recommend considering AWS Regions in the United States, Europe, or Asia Pacific, as appropriate for your latency and data residency requirements.
AddedUpdateupdate#18
Before: not present
After: We are providing an update on the ongoing service disruption. The Middle East (Bahrain) Region (ME-SOUTH-1) has suffered damage due to the conflict in the Middle East and is currently unavailable. Customers should recover their resources in other Regions from remote backups. Relevant billing operations are currently suspended while we restore normal operations in this AWS Region. This process is expected to take several months.
AddedUpdateupdate#19
Before: not present
After: We are providing an update on the disruption affecting the Middle East (Bahrain) (me-south-1) Region. The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand. After a thorough assessment, we have determined that we are unable to restore access to the resources and data hosted exclusively in this Region. When the first Availability Zone was damaged in March, we began recommending that customers migrate their workloads to other Regions, and most did so before the Region became unavailable following the disruption of a second Availability Zone in April. Since then, we have supported the remaining customers in re-establishing their operations in alternate Regions, using backups where available or implementing alternative solutions to mitigate the impact of inaccessible data. In parallel, we assessed all affected infrastructure and exhausted every option for restoring data and resources that had not been migrated before the Region became unavailable. We remain committed to supporting our customers in Bahrain in the long term and will share a further update in early 2027. We have notified the relevant authorities and continue to work with them toward that goal.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

Monday, March 2

1 incident change

All Amazon Web Services changesAmazon Web Services profileMarkdownJSON