Why did Amazon EC2 Auto Scaling terminate an instance?

7 minute read
0

My Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling group terminated an instance. However, I don't see a reason for the termination in the Amazon EC2 console.

Short description

Amazon EC2 Auto Scaling relies on Amazon EC2 or Elastic Load Balancing (ELB) health checks to determine the health state of an instance. You can view all the scaling actions of an Auto Scaling group, including health check replacements, in Activity History in the Amazon EC2 console.

Resolution

To determine your troubleshooting steps, use the instance's description in the Activity History view.

First, find the description and cause for the instance's termination:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, under Auto Scaling, choose Auto Scaling Groups. Then, select your instance's group.
  3. Choose the Activity view, and then search for the instance termination event in the Activity History.
  4. Note the Description and Cause for the instance termination event.

To troubleshoot the reason for your instance's termination, see the following Description examples.

"An instance was taken out of service in response to an EC2 health check indicating it has been terminated or stopped"

Amazon EC2 Auto Scaling terminates stopped and rebooted instances. To determine if a user manually stopped or rebooted the instance, check AWS CloudTrail history:

  1. Open the AWS CloudTrail console.
  2. In the navigation pane, choose Event history.
  3. Check the history for TerminateInstances, StopInstances, or RebootInstances API calls.

"An instance was taken out of service in response to an EC2 instance status checks failure"

Find instance or system check failures with Amazon CloudWatch metrics:

  1. Open the Amazon CloudWatch console.
  2. In the navigation pane, choose Metrics, and then choose the All Metrics view.
  3. Choose EC2 in the metrics panel, and then choose Per-Instance Metrics.
  4. Type the instance-id, and then choose StatusCheckFailed_Instance, StatusCheckFailed_System, or StatusCheckFailed to view metrics graphs.

"An instance was taken out of service in response to a ELB system health check failure"

If you turned on Elastic Load Balancing health checks for your Auto Scaling group, Amazon EC2 Auto Scaling performs both Elastic Load Balancing and EC2 health checks. Then, the service terminates instances that fail either health check. Verify that your Auto Scaling group has Elastic Load Balancing health checks activated:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then choose the instance's group.
  3. Choose the Details view, and then note the Health Check Type.
  4. If the health check type is "EC2, ELB" then the Auto Scaling Group has Elastic Load Balancing health checks enabled.

For more information, see An instance was taken out of service in response to an Elastic Load Balancing system health check failure.

In some cases, an Auto Scaling group has multiple attached target groups or load balancers. In this case, for Amazon EC2 Auto Scaling to consider an instance healthy, all target groups and load balancers must report it as healthy.

  1. In the navigation pane, under Auto Scaling, choose Auto Scaling Groups.
  2. Choose the instance's group, and then choose the Details view.
  3. Note if there is more than one load balancer attached under Load Balancers or Target Groups.
  4. Select the Target Group or Load Balancer.
  5. In the navigation pane, under Load Balancing, choose Target Groups, select your Target Group, and then choose Monitoring.
  6. To confirm if an instance is failing health checks, check the Healthy and Unhealthy Hosts metric graphs.

Check CloudTrail history to determine if a suspended process delayed the termination of an unhealthy instance until the process resumed:

  1. Open the AWS CloudTrail console.
  2. In the navigation pane, choose Event history.
  3. Check the history for SuspendProcesses and ResumeProcesses API calls.

"An instance was taken out of service in response to an EC2 Spot Instance interruption notice."

Amazon EC2 Auto Scaling terminates Spot Instances when either of the following conditions occur:

  • Capacity is no longer available.
  • Spot price exceeds the maximum price that you specified for the instances.

Activity History might show that the instance was removed from service because of a health check. To verify the termination reason, check the Spot requests status:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, under Instances, choose Spot Requests.
  3. Select the Spot request, choose the Description view, and then note the Status.

"An instance was taken out of service in response to a user request"

Review the CloudTrail events history for the period of time that the instance was taken out of service for any TerminateInstanceInAutoScalingGroup API calls:

  1. Open the AWS CloudTrail console.
  2. In the navigation pane, choose Event history.
  3. Check the history for TerminateInstanceInAutoScalingGroup API calls.

"A user request update of AutoScalingGroup constraints to min: 0, max: 2, desired: 1 changing the desired capacity from 2 to 1"

You can manually change the constraints of an Auto Scaling group. For instance, you can reduce its capacity. When you change a constraint, Amazon EC2 Auto Scaling might terminate instances to match the new settings.

"Instances were launched to balance instances in zones us-east-1a with other zones"

By default, Amazon EC2 Auto Scaling balances instances across all Availability Zones. When you add a new Availability Zone to an autoscaling group, Amazon EC2 Auto Scaling launches a new instance in that zone. Any rebalancing might terminate instances in other zones.

"An instance was taken out of service to balance the distribution of On-Demand and Spot capacities."

When an On-Demand or Spot Instance's distribution changes, the associated autoscaling group rebalances to match your desired purchase option distribution. Therefore, On-Demand or Spot Instances might terminate so that the autoscaling group can replace them to achieve your desired purchase option.

"A monitor alarm XXX-High-CPU-Utilization in state ALARM triggered policy AAA-scaledown changing the desired capacity from 2 to 1"

Amazon EC2 Auto Scaling can terminate instances in a group in response to a configured CloudWatch alarm. Check the group policies and CloudWatch alarm history.

Check the auto scaling group policies:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, under Auto Scaling, choose Auto Scaling Groups.
  3. Choose the instance's group.
  4. To view and edit policies, choose the Automatic Scaling pane

View the CloudWatch Alarm history:

  1. Open the Amazon CloudWatch console.
  2. In the navigation pane, choose Alarms.
  3. Choose the alarm, and then choose the History view.
  4. Check the history for any state changes to the alarm or modifications to the alarm configuration.

"An instance was taken out of service in response to a user health-check"

You can define custom health checks in Amazon EC2 Auto Scaling. When a custom health check determines that an instance is unhealthy, the check manually starts SetInstanceHealth. Then, it sets the instance's state to Unhealthy. Amazon EC2 Auto Scaling terminates the unhealthy instance on its next run.

"A scheduled action update of AutoScalingGroup constraints to min: 1, max: 9, desired: 1 changing the desired capacity from 2 to 1"

You can configure scheduled actions that change an Auto Scaling group's minimum, maximum, or desired capacity. When the scheduled action decreases the desired capacity, Amazon EC2 Auto Scaling terminates one or more instances to match the new desired capacity.

Related information

Why did my Auto Scaling group scale down?

Why didn't Amazon EC2 Auto Scaling terminate an unhealthy instance?

AWS OFFICIAL
AWS OFFICIALUpdated 8 months ago
No comments