rewindex

ChangesAmazon Web ServicesElevated API issues and Launch Failures (ap-northeast-1)

Amazon Web Services: Incident silently edited: Elevated API issues and Launch Failures (ap-northeast-1)

Silent editminor edit (typesafe/jev-1.13-20260917, 54%)Changed health.aws.amazon.com (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what Amazon Web Services removed; blue is what they added. Muted text is unchanged context.

ChangedUpdateresolved#1
Before: 日本時間 2026 年 11 月 11 日 午後 11 時 46 分から 2026 年 11 月 12 日 午前 2 時 46 分 (太平洋標準時 午前 6 時 46 分から午前 9 時 46 分) にかけて、主に単一のアベイラビリティーゾーン (apne1-az1) において、API エラー率の上昇および起動失敗が発生しておりました。また、ECS および Fargate のタスク起動失敗、ELB のプロビジョニング遅延、NAT Gateway の作成および関連付け解除ワークフローの失敗、Managed Flink アプリケーションの状態遷移の遅延など、他サービスにも影響が発生しておりました。稼働中の EC2 インスタンスおよび既存のワークフローには影響はございませんでした。この時間帯において、影響を受けているアベイラビリティーゾーンでは EC2 および EBS のヘルスデータが欠損または遅延し、CloudWatch アラームが INSUFFICIENT_DATA 状態に移行し、DescribeInstanceStatus API が Insufficient Data を返しておりました。 根本原因を特定し、日本時間 2026 年 11 月 12 日 午前 0 時 47 分までに緩和策の実施を開始しました。日本時間 午前 2 時 46 分には API の広範な回復が見られ、新規 EC2 インスタンスの起動が成功するようになりました。完全な復旧を監視する中で、影響を受けた単一のアベイラビリティーゾーンにおける新規起動インスタンスで接続の問題が発生していることを確認し始めました。インスタンス間の接続、および影響発生前に起動されたインスタンスへの接続は影響を受けておりませんでした。この問題が最初の問題と同一の根本原因であることを特定することができました。 接続の問題を軽減するために複数の対策を並行して実施し、日本時間 午前 4 時 51 分に接続の問題の回復の最初の兆候を確認し、日本時間 午前 5 時 31 分に完全に回復いたしました。問題は解決され、サービスは正常に稼働しております。 | Between 6:46 AM and 9:46 AM PST, we experienced elevated API error rates and launch failures primarily centered in a single Availability Zone (apne1-az1). We also experienced impact to other services such as ECS and Fargate Task Launch failures, ELB provisioning delays, Nat Gateway failures … EC2 and EBS health data was missing or…
After: Between 6:46 AM PST and 12:31 PM PST, we experienced periods of API error rates, as well as connectivity issues for newly launched EC2 instances, primarily affecting a single Availability Zone (apne1-az1) in the AP-NORTHEAST-1 Region. Engineers were automatically engaged and immediately began investigating. The first impact window was between 6:46 AM PST and 9:46 AM PST, and primarily manifested as API errors and failed EC2 instance launches. Other AWS Services that rely on the EC2 APIs for their workflows, such as Autoscaling, ECS, and RDS, were also affected by this issue. Existing EC2 instances were not affected. During this first impact period, customers may have experienced API throttling or 5XX Responses. Additionally, some EC2 RunInstance API calls may have succeeded, but instances did not successfully launch, instead transitioning to a TERMINATED state. EC2 and EBS health data was missing or…
Before: …EC2 and EBS health data was missing or delayed for the affected Availability Zone resulting in CloudWatch alarms to transition to INSUFFICENT_DATA state and the DescribeInstanceStatus API to return Insufficient Data. We had identified the root cause and began our mitigation efforts by 7:47 AM, and by 9:46 AM we were seeing broad recovery for APIs and were able to see new EC2 launches succeed. While monitoring for full recovery, we began to observe that new launches in the single affected Availability Zone were experiencing connectivity issues, instance to instance connectivity was unaffected as was instances launched before the impact began. We were able to identify that this shared a root cause to the initial issue. We took multiple parallel paths to mitigate the connectivity issue, and saw first signs of recovery for the connectivity issue at 11:51 AM and full recovery by 12:31 PM. The issue has been resolved and all services…
After: …EC2 and EBS health data was missing or delayed, which resulted in CloudWatch alarms transitioning to an INSUFFICENT_DATA state. We recommend customers retry failed API requests or workflows that failed for any of the affected services. At 9:46 AM PST, we mitigated the API errors, but had not yet mitigated the underlying root cause. While monitoring recovery of the EC2 APIs, we observed that both newly launched EC2 instances and existing instances where the network configuration was modified were experiencing connectivity issues. These connectivity issues were the result of delays in an internal subsytem responsible for propagating EC2 instance networking changes, and related to the underlying root cause. By 12:31 PM PST, we had fully resolved the issue. During this time, we were working on multiple parallel paths to both mitigate the connectivity issues and fully understand the root cause. Other AWS Services that launch EC2 instances or modify EC2 instance networking configuration, such as ECS, EKS, ELB, and NAT Gateway, were also affected by this issue. A list of impacted services is available below. The issue has been resolved and all services…
Before: …available below. The issue has been resolved and the service is operating normally.…
After: …available below. The issue has been resolved and all services are operating normally.…

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Rewindex captured a newer version after the last checkpoint; the next run () commits it.

History of this incident

TodayThursday, November 12Thu, Nov 12

3 rewrites, 1 incident change

YesterdayWednesday, November 11Wed, Nov 11

7 rewrites, 10 incident changes

Cite this

Incident silently edited: Elevated API issues and Launch Failures (ap-northeast-1) on Amazon Web Services’s status page, changed 2026-11-12 01:53 UTC (change 18 of 21).

Title
Incident silently edited: Elevated API issues and Launch Failures (ap-northeast-1)
Vendor
Amazon Web Services
Date
URL
https://rewindex.com/change/r3b2e59886f59df

All Amazon Web Services changesAmazon Web Services profileMarkdownJSON