How can I configure EC2 instances in an Auto Scaling group that has no internet access to send metrics and logs to CloudWatch?

Last updated: 2020-11-09

How can I configure Amazon Elastic Compute Cloud (Amazon EC2) instances in an Amazon EC2 Auto Scaling group that has no internet access to send logs and metrics to Amazon CloudWatch?

Resolution

  1. Install the CloudWatch Agent on an Amazon EC2 instance. This instance must have internet connectivity. Or, you can choose an Amazon EC2 instance that's already pushing the logs and metrics to CloudWatch using the CloudWatch agent.
  2. Verify that the CloudWatch agent is pushing metrics and logs from your Amazon EC2 instance.
  3. Create a launch template for the Auto Scaling group. To enable instances to push the metrics and logs to CloudWatch, provide the correct AWS Identity and Access Management (IAM) role in the launch template. In the advanced settings, define the user data with a script similar to the following. Use a customized version of this script to install and configure the CloudWatch agent based on the JSON configuration from step 1.
    Note: This example user data script installs the CloudWatch agent in an Amazon EC2 Linux instance. The script configures the agent to monitor memory and disk utilization, and then starts the agent. You must use the download link for the specific Region of your Auto Scaling group.
#!/bin/bash
cd /tmp
wget https://s3.<region>.amazonaws.com/amazoncloudwatch-agent-<region>/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
cat << EOF > /opt/aws/amazon-cloudwatch-agent/bin/config.json
{
  "agent": {
    "metrics_collection_interval": 60
  },
  "metrics": {
    "append_dimensions": {
      "InstanceId": "\${aws:InstanceId}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "disk_used_percent"
        ],
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      }
    }
  }
}
EOF
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

  • Add interface virtual private cloud (VPC) endpoints for CloudWatch monitoring and CloudWatch Logs to the VPC that hosts the private subnets. To find the correct endpoint, see Amazon CloudWatch endpoints and quotas. To allow only the required CloudWatch actions in these VPC endpoints, update the endpoint policies with custom policies.

    Example of a policy for the CloudWatch monitoring VPC endpoint:

  • {
      "Statement": [
        {
          "Sid": "PutOnly",
          "Principal": "*",
          "Action": [
            "cloudwatch:PutMetricData"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
      ]
    }

    Example of a policy for the CloudWatch Logs VPC endpoint:

    {
      "Statement": [
        {
          "Sid": "PutOnly",
          "Principal": "*",
          "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
      ]
    }

    Refer to "Considerations for creating an interface VPC endpoint" later for more details.

  • Add a virtual private cloud (VPC) gateway endpoint for Amazon S3 to the VPC that hosts the private subnets. This endpoint allows the user data script from the instances in the private subnets to access and download the CloudWatch agent package from Amazon S3.
  • Create an Auto Scaling group (with private subnets enabled) using the launch template you created in step 3. The CloudWatch agent runs in the instances that you launch in this Auto Scaling group. The agent also sends metrics and logs through the VPC interface endpoints that you created in step 4.
  • Considerations for creating an interface VPC endpoint

    • Be sure to use the endpoint that corresponds with the AWS Region of your Auto Scaling group. For example, if the Auto Scaling group is in the London Region, the endpoint for metrics is monitoring.eu-west-2.amazonaws.com. The endpoint for logs in this scenario is logs.eu-west-2.amazonaws.com.
    • Confirm that you’ve enabled the Enable Private DNS name option. You can enable this option only if the Enable DNS hostnames and Enable DNS Support attributes are set to true for the VPC. If this option is disabled, the VPC interface endpoint isn't mapped to the service endpoint. As a result, the instances can't reach the public service endpoint. Enabling this option maps the service endpoint to the VPC interface endpoint and makes communication to the service endpoint private. By default, the CloudWatch agent connects to this endpoint. You can use the endpoint_override parameter in the agent configuration file to override the default endpoint, if required.
    • Confirm that the security group rules allow communication between the endpoint network interface and the resources in your VPC that communicate with the service. The API calls for pushing the logs and metrics are HTTPS-based GET/POST requests. The endpoint network interface security group requires inbound rules for HTTPS protocol from the source IPs. The source IP addresses are the IP addresses of the EC2 instances pushing the metrics and logs, or the VPC CIDR.
    • When instances are part of an Auto Scaling group, specify one of the dimensions as the Auto Scaling group name in the agent configuration file. To find the name of the Auto Scaling group, the agent gets the tags associated with the instance from the Amazon EC2 endpoint. You must add the VPC interface endpoint for the Amazon EC2 service. The agent gets the ImageId, InstanceId, and InstanceType values from the Amazon EC2 instance's metadata.

    • Did this article help?


      Do you need billing or technical support?