My Amazon EC2 instance is marked as unhealthy, moved to the Auto Scaling Terminating state, and eventually terminated before I can find the cause of the problem.

You can add a lifecycle hook to your Auto Scaling group to move instances in the Terminating state to the Terminating:Wait state. This state allows you to access these instances before they are terminated, permitting you to troubleshoot why they were being marked unhealthy.

By default, an instance remains in the Terminating:Wait state for 3600 seconds (1 hour). To increase this time, you can use the heartbeat-timeout parameter in the put-lifecycle-hook API call. The maximum amount of time that you can keep an instance in the Terminating:Wait state is 48 hours or 100 times the heartbeat timeout, whichever is smaller.

Lifecycle hooks can only be configured using the AWS CLI and API. The following steps describe the process of configuring a lifecycle hook, including creating the necessary SNS topic and IAM permissions using the AWS CLI. If you prefer to manage IAM and SNS by using the console, see Create Role for IAM and Create a Topic for SNS.

1.    Create a SNS topic for Auto Scaling to send lifecycle notifications to. The following CLI example calls the SNS create-topic command to create the SNS topic ASNotifications:

$ aws sns create-topic --name ASNotifications

2.    Save the Amazon Resource Name (ARN) that is returned. The ARN returned should resemble the following:

"TopicArn": "arn:aws:sns:us-west-2:123456789012:ASNotifications"

3.    Create a subscription to the SNS topic. This is required to receive the LifecycleActionToken that is needed to either extend the heartbeat timeout of the pending state or to complete the lifecycle action. The following example uses the aws sns subscribe command to create a subscription that uses the email protocol (SMTP) with the endpoint email address user@amazon.com.

$ aws sns subscribe --topic-arn arn:aws:sns:us-west-2:123456789012:ASNotifications --protocol email --notification-endpoint user@amazon.com

IAM permissions are configured by creating an IAM role that grants the Auto Scaling service permissions to send to the SNS topic. This can be accomplished by creating a text file that contains the appropriate policy and then referencing the file from the aws iam create-role command. The following steps show how to do this.

1.    Use a text editor such as vi to create the text file:

$ sudo vi assume-role.txt

2.    Paste the following contents into the text file and save the file.

{
  "Version": "2012-10-17",
  "Statement": [{
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "autoscaling.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

3.    The following example uses the aws iam create-role command to create the IAM role AS-Lifecycle-Hook-Role from the policy saved to assume-role.txt:

$ aws iam create-role --role-name AS-Lifecycle-Hook-Role --assume-role-policy-document file://assume-role.txt

The output contains the ARN for the role. Be sure to save both the ARN of the IAM role and the SNS topic.

4.    Add permissions to the role to allow Auto Scaling to send SNS notifications when a lifecycle hook event occurs. The following example uses the aws iam attach-role-policy command to attach the managed policy AutoScalingNotificationAccessRole to the IAM role AS-Lifecycle-Hook-Role:

$ aws iam attach-role-policy --role-name AS-Lifecycle-Hook-Role --policy-arn arn:aws:iam::aws:policy/service-role/AutoScalingNotificationAccessRole

This managed policy grants the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl",
        "sns:Publish"
      ]
    }
  ]
}

A lifecycle hook can be configured after notifications and permissions have been properly configured. The following example uses the aws autoscaling put-lifecycle-hook command to configure the lifecycle hook:

aws autoscaling put-lifecycle-hook --lifecycle-hook-name AStroublshoot --auto-scaling-group-name MyASGroup
         --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING
         --notification-target-arn arn:aws:sns:us-west-2:123456789012:ASNotifications
         --role-arn arn:aws:iam::123456789012:role/AS-Lifecycle-Hook-Role

This command performs the following tasks:

  • Names the lifecycle hook (AStroubleshoot).
  • Identifies the Auto Scaling group that the lifecycle hook is associated with (MyASGroup).
  • Configures the hook for the instance termination lifecycle stage (EC2_INSTANCE_TERMINATING).
  • Specifies the SNS topic ARN (arn:aws:sns:us-west-2:123456789012:ASNotifications).
  • Specifies the IAM role ARN (arn:aws:iam::123456789012:role/AS-Lifecycle-Hook-Role).

Be sure to substitute your own Auto Scaling group name, SNS target ARN, and IAM role ARN where appropriate before running this command in your environment.

To test the lifecycle hook, choose an instance and use terminate-instance-in-auto-scaling group to terminate the instance. This makes Auto Scaling terminate the instance, similar to what it does when the instance becomes unhealthy. After the instance moves to the Terminating:Wait state, you can choose to keep your instance in the Terminating:Wait state using record-lifecycle-action-heartbeat or allow the termination to be completed by using complete-lifecycle-action. The following example provides syntax for doing this:

aws autoscaling complete-lifecycle-action --lifecycle-hook-name my-lifecycle-hook
         --auto-scaling-group-name MyASGroup --lifecycle-action-result CONTINUE
         --instance-id i-0e7380909ffaab747

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center.

Published: 2015-08-04

Updated: 2017-08-25