How do I troubleshoot issues with including metadata on an EC2 instance in CloudFormation?

Last updated: 2022-04-12

I used AWS::CloudFormation::Init to include metadata on an Amazon Elastic Cloud Compute (Amazon EC2) instance, but I don't see the changes on the instance.

Short description

Issues with EC2 instance metadata in an AWS CloudFormation stack can occur due to the following reasons:

  • The cfn-init helper script isn't installed on one or more instances of the CloudFormation stack. To resolve this issue, complete the steps in the Verify that the cfn-init helper script is installed section.
  • The instance isn't connected to the internet. To resolve this issue, complete the steps in the Verify that the instance is connected to the internet section.
  • The CloudFormation template contains syntax errors or incorrect values. To resolve this issue, complete the steps in the Search for errors in the cloud-init or cfn-init logs section.

Important: Before you complete the following resolutions, set the Rollback on failure option for your CloudFormation stack to No.

Note: The following resolutions are specific to CloudFormation stacks that are created with Linux instances.

Resolution

Verify that the cfn-init helper script is installed

To confirm that cfn-init is installed on the instance that's configured to send signals to CloudFormation resources:

1.    Connect to the instance using SSH.

2.    Run one of the following commands to verify that cfn-init or the aws-cfn-bootstrap package is installed in your directory.

cfn-init:

$ sudo find / -name cfn-init
/opt/aws/bin/cfn-init
/opt/aws/apitools/cfn-init
/opt/aws/apitools/cfn-init-1.4-34.24.amzn1/bin/cfn-init
/var/lib/cfn-init

-or-

aws-cfn-bootstrap package:

$ sudo rpm -q aws-cfn-bootstrap
aws-cfn-bootstrap-1.4-34.24.amzn1.noarch

Important: The preceding command works only on distributions that use the RPM Package Manager.

Note: By default, CloudFormation helper scripts are installed on the Amazon Linux Amazon Machine Image (AMI). If CloudFormation helper scripts aren't installed, see CloudFormation helper scripts reference for instructions on how to install them.

Verify that the instance is connected to the internet

If the instance is in an Amazon Virtual Private Cloud (Amazon VPC), then it can connect to the internet through:

  • A NAT device in a private subnet
  • An internet gateway in a public subnet

To test the instance's internet connection, access a public webpage, such as AWS, and run a curl command on the instance. For example:

curl -I https://aws.amazon.com

Note: If the instance is connected to the internet, then the command returns an HTTP 200 status code.

If you're using an interface VPC endpoint, then the endpoint must be located in the same AWS Region as the instance. Also, the security group that's attached to the interface endpoint must allow incoming connections on port 443 from the private subnet of the Amazon VPC.

Search for errors in the cloud-init or cfn-init logs

To search for errors in the cloud-init logs or cfn-init logs:

1.    Connect to your instance using SSH.

2.    Look for detailed error or failure messages by searching for the keywords "error" or "failure" in the following logs:

  • /var/log/cloud-init-output.log
  • /var/log/cloud-init.log
  • /var/log/cfn-init.log
  • /var/log/cfn-init-cmd.log

To parse all instances of the words "error" or "failure" in /var/log/cfn or /var/log/cloud-init files, run the following command:

grep -ni 'error\|failure' $(sudo find /var/log -name cfn-init\* -or -name cloud-init\*)

Note: The preceding command returns the file name, line number, and error message.

Look for cfn-init.log. If you can't find it, then cfn-init wasn't run. You must also check cloud-init-output.log and cloud-init.log to see if there was a failure when running user data. After you identified the error, fix it based on the error message, and then recreate the stack.

If cfn-init.log exists, then cfn-init was run, but a failure occurred. Check cfn-init.log to see what went wrong, and fix it based on the error message.

To confirm that the UserData property is configured to run cfn-init, complete the following steps:

  1. In a code editor, open the AWS CloudFormation template for your stack, and then find the UserData property section.
  2. Check for errors, including syntax errors, missing spaces, misspellings, and other typos.
  3. Confirm that the values for the stack, resource, and Region properties are correct.
  4. For the Fn::Join intrinsic function of the UserData property, use the -v option to run cfn-init in verbose mode. See JSON and YAML example templates.