Manju helps you understand
why Auto Scaling might terminate
a healthy instance

Manju_DUB

My Amazon EC2 Auto Scaling group terminated an instance, but I don't see a reason for the termination on the EC2 console. How do I find out why Amazon EC2 Auto Scaling terminated my instance?

Amazon EC2 Auto Scaling relies on Amazon Elastic Compute Cloud (EC2) or Elastic Load Balancing (ELB) health checks to determine the health state of an instance. All the scaling actions of an Auto Scaling group, including health check replacements, are visible in the Activity History view on the EC2 console.

Use the instance’s description in the Activity History view to determine further steps.

Before proceeding, find the description and cause for the instance’s termination:

  1. On the Amazon EC2 console navigation pane under Auto Scaling, choose Auto Scaling Groups, and then choose the instance’s group.
  2. Choose the Activity History view, and then choose the instance termination event.
  3. Note the Description and Cause for the instance termination event.

Refer to the following description examples to understand probable underlying reasons for instance termination:

"An instance was taken out of service in response to an EC2 health check indicating it has been terminated or stopped."

Find instance or system check failures with Amazon CloudWatch metrics:

  1. On the CloudWatch console navigation pane, choose Metrics, and then choose the All Metrics view.
  2. Choose EC2 in the metrics panel, and then choose Per-Instance Metrics.
  3. Type the instance-id, and then choose StatusCheckFailed_Instance or StatusCheckFailed_System or StatusCheckFailed to view metrics graphs.

Amazon EC2 Auto Scaling terminates stopped and rebooted instances. Check AWS CloudTrail history to determine if a user manually stopped or rebooted the instance:

  1. On the CloudTrail console navigation pane, choose Event history.
  2. Check the history for TerminateInstances, StopInstances, or RebootInstances API calls.

Amazon EC2 Auto Scaling terminates Spot Instances when the Spot price exceeds the bid price. Activity History might show that the instance was removed from service due to a health check. Verify the termination reason in the price history:

  1. On the Amazon EC2 console navigation pane, under Instances, choose Spot Requests.
  2. Choose Pricing History.
  3. Filter the history by product (operating system), instance type, date range, and Availability Zone to see the instance’s price history.

If an Auto Scaling group’s health check type is set to ELB, Amazon EC2 Auto Scaling performs both ELB and EC2 health checks and terminates instances that fail either health check:

  1. On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then choose the instance's group.
  2. Choose the Details view, and then note the Health Check Type.

"An instance was taken out of service in response to a ELB system health check failure."

When multiple load balancers are attached to an Auto Scaling group, all load balancers must report an instance is healthy for Amazon EC2 Auto Scaling to consider the instance healthy:

  1. On the Amazon EC2 console navigation pane, under Load Balancing, choose Load Balancers, then choose Monitoring.
  2. Select the Health Hosts Count metric graph to confirm that the instance is failing health checks.
  3. On the navigation pane, under Auto Scaling, choose Auto Scaling Groups.
  4. Choose the instance’s group and choose the Details view.
  5. Note if there is more than one load balancer attached under Load Balancers or Target Groups.

Check CloudTrail history to determine if a suspended process delayed the termination of an unhealthy instance until the process resumed:

  1. On the CloudTrail console navigation pane, choose Event history.
  2. Check the history for SuspendProcesses and ResumeProcesses API calls.

"an instance was taken out of service in response to a user health-check"

You can define custom health checks in Amazon EC2 Auto Scaling. When a custom health check determines that an instance is unhealthy, the check manually triggers SetInstanceHealth and then sets the instance's state to Unhealthy. Amazon EC2 Auto Scaling terminates the unhealthy instance on its next run.

"An instance was launched to aid in balancing the group's zones" or "Instances were launched to balance instances in zones us-east-1a with other zones"

By default, Amazon EC2 Auto Scaling balances instances across all Availability Zones. When a user adds a new Availability Zone to an Auto Scaling group, Amazon EC2 Auto Scaling launches a new instance in that zone, and rebalancing might terminate instances in other zones.

"At 2018-02-12T13:48:46Z a monitor alarm XXX-High-CPU-Utilization in state ALARM triggered policy AAA-scaledown changing the desired capacity from 2 to 1"

Amazon EC2 Auto Scaling can terminate instances in a group in response to a configured CloudWatch alarm. Check the group policies and CloudWatch alarm history:

To check the Auto Scaling group policies:

  1. On the Amazon EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups.
  2. Choose the instance’s group.
  3. Choose the Scaling Policies pane to view and edit policies.

To view the CloudWatch Alarm history:

  1. On the CloudWatch console navigation pane, choose Alarms.
  2. Choose the alarm, and then choose the History view.
  3. Check the history for any state changes to the alarm or modifications to the alarm configuration.

"At 2018-02-12T13:49:12Z a scheduled action update of AutoScalingGroup constraints to min: 1, max: 9, desired: 1 changing the desired capacity from 2 to 1. [...]"

Users can configure scheduled actions that change an Auto Scaling group’s minimum, maximum, or desired capacity. When the scheduled action decreases the desired capacity, Amazon EC2 Auto Scaling terminates one or more instances to match the new desired capacity, such as in this example.

"a user request update of AutoScalingGroup constraints to min: 0, max: 2, desired: 1 changing the desired capacity from 2 to 1"

When a user manually changes the constraints of an Auto Scaling group, such as reducing the desired capacity, Amazon EC2 Auto Scaling can terminate instances to match the new settings.  


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2018-06-13