How can I configure EC2 instances in an Auto Scaling group that has no internet access to send metrics and logs to CloudWatch?

Resolution

1.    Install the CloudWatch agent on your Amazon EC2 instance. This instance must have internet connectivity. Or, you can choose an EC2 instance that's already pushing the logs and metrics to CloudWatch, using the CloudWatch agent.

2.    Verify that the CloudWatch agent is pushing metrics and logs from your EC2 instance.

3.    Create a launch template for the Auto Scaling group. In your launch template, complete the following steps in the advanced settings:
For IAM instance profile, select the relevant correct AWS Identity and Access Management (IAM) role that allows instances to push the metrics and logs to CloudWatch.
For user data, enter a script that's similar to the following example. Use a customized version of this script to install and configure the CloudWatch agent based on the JSON configuration from step 1**:
Note:** The following user data script installs the CloudWatch agent in an EC2 Linux instance. The script configures the agent to monitor memory and disk utilization, and then starts the agent. You must use the download link for the specific AWS Region of your Auto Scaling group.

#!/bin/bash
cd /tmp
wget https://s3.<region>.amazonaws.com/amazoncloudwatch-agent-<region>/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
cat << EOF > /opt/aws/amazon-cloudwatch-agent/bin/config.json
{
  "agent": {
    "metrics_collection_interval": 60
  },
  "metrics": {
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "disk_used_percent"
        ],
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      }
    }
  }
}
EOF
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

4.    Add interface virtual private cloud (VPC) endpoints for CloudWatch monitoring and Amazon CloudWatch Logs to the VPC that hosts the private subnets. To find the correct endpoint, see Amazon CloudWatch endpoints and quotas. To allow only the required CloudWatch actions in these VPC endpoints, update the endpoint policies with custom policies.

See the following example of a policy for the CloudWatch monitoring VPC endpoint:

{
  "Statement": [
    {
      "Sid": "PutOnly",
      "Principal": "*",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

See the following example of a policy for the CloudWatch Logs VPC endpoint:

{
  "Statement": [
    {
      "Sid": "PutOnly",
      "Principal": "*",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

For more details, refer to the Considerations for creating an interface VPC endpoint section.

5.    Add a virtual private cloud (VPC) gateway endpoint for Amazon Simple Storage Service (Amazon S3) to the VPC that hosts the private subnets. This allows the user data script from the instances in the private subnets to access and download the CloudWatch agent package from Amazon S3.

6.    Create an Auto Scaling group (with private subnets activated) using the launch template that you created in step 3. The CloudWatch agent runs in the instances that you launch in this Auto Scaling group. The agent also sends metrics and logs through the VPC interface endpoints that you created in step 4.

Considerations for creating an interface VPC endpoint

• Be sure to use the endpoint that corresponds with the Region of your Auto Scaling group. For example, if the Auto Scaling group is in the London Region, then the endpoint for metrics is monitoring.eu-west-2.amazonaws.com. The endpoint for logs in this scenario is logs.eu-west-2.amazonaws.com.
• Confirm that you’ve turned on the Enable Private DNS name option. You can turn on this option only if the Enable DNS hostnames and Enable DNS Support attributes are set to true for the VPC. If this option is turned off, then the VPC interface endpoint isn't mapped to the service endpoint. As a result, the instances can't reach the public service endpoint. Turning on this option maps the service endpoint to the VPC interface endpoint and makes communication to the service endpoint private. By default, the CloudWatch agent connects to this endpoint. You can use the endpoint_override parameter in the agent configuration file to override the default endpoint, if required.
• Confirm that the security group rules allow communication between the endpoint network interface and the resources in your VPC that communicate with the service. The API calls for pushing the logs and metrics are HTTPS-based GET/POST requests. The endpoint network interface security group requires inbound rules for HTTPS protocol from the source IPs. The source IP addresses are the IP addresses of the EC2 instances pushing the metrics and logs, or the VPC CIDR.
• When instances are part of an Auto Scaling group, specify one of the dimensions as the Auto Scaling group name in the agent configuration file. To find the name of the Auto Scaling group, the agent gets the tags associated with the instance from the Amazon EC2 endpoint. You must add the VPC interface endpoint for the Amazon EC2 service. The agent gets the ImageId, InstanceId, and InstanceType values from the EC2 instance's metadata.