Automate AWS Systems Manager activation for hybrid-managed node registration

AWS Systems Manager (formerly known as SSM) is an AWS service that you can use to view and control your servers on AWS cloud and on-premises infrastructure. Systems Manager makes it easy to manage a hybrid environment.

To set up servers and virtual machines (VMs) in your hybrid environment as Systems Manager managed instances, you create a managed-instance activation. Creating and managing Systems Manager Hybrid Activations credentials for your on-premises servers and VMs can be a manual and tedious task. The Hybrid Activations credentials can reach its activation expiration date or registration limit value after which the credentials can no longer be used to register the servers. The credentials will need to be recreated manually and the new servers have to be configured to use this new credentials. The core ask is to automate the creation and management of the System Manager Hybrid Activations credentials, reducing the operational support needed in this task.

In this post, I will walk through the solution on automating the System Manager Hybrid Activations creation.

Solution overview

The solution is enabled using AWS CloudFormation stack. The Cloudformation stack creates the AWS resources on your account needed for the solution. These resources are as follows:

• Amazon API Gateway: REST API of Private Type, Integrated with AWS Lambda function. When the web client from the on-premises server performs a GET request to the API gateway, it returns the Hybrid Actions Code/ID combination.
• AWS Lambda: The Lambda function provides the Hybrid Activations Code/ID combination to the on-premises server via the API gateway. It will create a new activation code if it finds the existing activation code is expired or has reached registration limit.
• Amazon DynamoDB: To store the state, the Lambda updates the table to the ‘Locked’ state if it’s serving a request from a client. It updates the table to ‘Unlocked’ after completing serving the request.
• Amazon VPC Endpoint: VPC Endpoint for API gateway for privately accessing the API gateway URL from the on-premises network.
• AWS Systems Manager Parameter Store: To store the Hybrid Activations ID/Code.

Solution Overview

The following is a brief flow of the executions:

1.  The web client calls the private API Gateway endpoint (for example, GET).
2.  When connecting from on-premises servers, the on-premises DNS server should be configured to forward requests to VPC DNS to get the private IP address of the VPC Endpoint. The DNS server resolves and sends back the IP address to the web client.
3. The request is sent to the private IP address of the VPC Endpoint of the API Gateway.
4. The resource Policy of the API gateway is checked to see if the request is coming from the VPC endpoint of the API gateway. If not, then it’s forbidden.
5. API Gateway passes the request to Lambda through an integration request.
6. Lambda updates the state key in DynamoDB to ‘Locked’, indicating it’s serving the request.
7. Lambda retrieves the credentials from the Parameter Store and sends it back to the client.

Walkthrough

Prerequisites

For this walkthrough, you should have the following:

• An AWS account
• An AWS Identity and Access Management (IAM) user/role who can:
  • Create a private API, create a method, and deploy it in API Gateway.
  • Create Lambda function, DynamoDB, Parameter Store Parameter, and AWS CloudWatch Log Group.
  • Create a new IAM role with a trust policy. Read more about Granting least privilege when creating IAM policies.
• The VPC to which you’re deploying must have both enableDnsSupport and enableDnsHostnames VPC attributes set to true.
• Basic familiarity with AWS CloudFormation, Systems Manager, and Amazon API Gateway.

Step1: Create VPC endpoint for API Gateway

In the first step, you create VPC endpoints for the API Gateway in your VPC. You also create a security group attached to the endpoint to allow a TCP port 443. Use the following steps to automate this using CloudFormation.

Note that if a VPC endpoint for API gateway already exists for the VPC, skip this step and note the existing VPC endpoint ID.

1. Download the CloudFormation Template.
2. Visit the AWS CloudFormation console in your preferred region.
3. Choose Create stack, and then choose With new resources (standard).
4. On the Create stack page, select Upload a template. Choose the template that you downloaded in the preceding step. Then, select Next.
5. Provide a Stack name. For example, apigateway-vpcendpoint-setup.
6. The CloudFormation stack requires a few parameters, as shown in the following screenshot:

Example for the Cloudformation Stack Parameters.

7. Choose Next on the Configure stack options page.
8. Review the configuration options and choose Create stack.
9. Verify that the stack has a status of CREATE_COMPLETE.
10. Once the stack has been created, refer the Outputs section of your stack and copy the VPC endpoint ID.

Step2: Create a KMS Key

In this step, you’ll create an AWS Key Management Service (AWS KMS) key to encrypt Parameter Store. Here, Parameter store is used to store the Activation Code and Activation ID. To create a KMS key:

1. Open the AWS KMS console here.
2. In the navigation pane, choose Customer managed keys and select Create Key.
3. Choose the symmetric AWS KMS key, and select Next
4. Review the other configuration options, and create the Key.
5. Once created, note the key ID.

Step3: Create API Gateway and Lambda

In the final step, you’ll create and deploy a Private API and Lambda function. Use the following steps to automate this using CloudFormation.

1. Download the CloudFormation Template.
2. Visit the AWS CloudFormation console in your preferred region.
3. Choose Create stack, and then choose With new resources (standard).
4. On the Create stack page, select Upload a template. Choose the template that you downloaded in the preceding step. Then, select Next.
5. Provide a Stack name. For example, apigateway-lambda-setup.
6. The CloudFormation stack requires a few parameters, as shown in the following screenshot:

Example for the Cloudformation Stack Parameters.

7. Review the details of your parameters, and check the box “I acknowledge that AWS CloudFormation might create IAM resources”. Then select Create stack to start building the resources.
8. Once the stack has been created, refer to the Outputs section of your stack and copy the API Gateway Invoke URL.

Cloudformation Stack Output

9. From the on-premises server, which needs to be registered, access the copied URL using curl/wget or any other web client. The Activation ID/Code combination is returned in the JSON format. In the following example, on my Linux terminal, I am using curl and an optional jq package command to give a structured and formatted view of the output.

[root@ec2amaz-r1rvyg ~]# curl -s https://o2h4ocy7q6.execute-api.us-east-1.amazonaws.com/lambdastage | jq '.'
{
  "ActivationId": "e50a8437-23dd-4326-9e79-5e3b7573493e",
  "ActivationCode": "vVcH9zJX4ROy2XTsh5cb"
}

Note that you should replace the URL in the example with the URL from your CloudFormation Stack Output.

You can improve the security of the private API created above by configuring the VPC endpoint to use VPC endpoint policy. A VPC endpoint policy is an IAM resource policy that you can attach to an interface VPC endpoint to control access to the endpoint. VPC endpoint policies can be used together with API Gateway resource policies. The resource policy is used to specify which principals can access the API. The endpoint policy specifies which private APIs can be called via the VPC endpoint.

Follow the documentation reference to Create VPC endpoint policies for private APIs in API Gateway

Example scripts for automatic activation

You can use the API Gateway Invoke URL that you copied from the output section of the CloudFormation stack in your Shell/PowerShell script when installing SSM Agent. For testing and validation, you can save and run the following example scripts on a Redhat Based server or a Windows Server. For deployment at scale, have the script run on your server launch.

Linux:

– A Shell script to retrieve Hybrid Activation credentials and install SSM Agent with the obtained credentials and register to the us-east-1 region:

#!/bin/bash
sudo yum erase amazon-ssm-agent --assumeyes

credentials=$(curl -s https://cla9phiczg.execute-api.us-east-1.amazonaws.com/lambdastage)
activationcode=$(echo $credentials | jq -r '.ActivationCode')
activationid=$(echo $credentials| jq -r '.ActivationId') 

mkdir /tmp/ssm
curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm -o /tmp/ssm/amazon-ssm-agent.rpm
sudo dnf install -y /tmp/ssm/amazon-ssm-agent.rpm
sudo systemctl stop amazon-ssm-agent
sudo -E amazon-ssm-agent -register -code $activationcode -id $activationid -region us-east-1
sudo systemctl start amazon-ssm-agent

Note that you should replace the URL in the example with the URL from your CloudFormation Stack Output.

Windows:

– A PowerShell script to retrieve Hybrid Activation credentials and install SSM Agent with the obtained credentials and register to the us-east-1 region:

$credential = Invoke-WebRequest -URI https://cla9phiczg.execute-api.us-east-1.amazonaws.com/lambdastage | Select-Object -ExpandProperty Content
$credentialPSObject = $credential | ConvertFrom-Json
 
$code = $credentialPSObject.ActivationCode
$id = $credentialPSObject.ActivationId
$region = "us-east-1"
$dir = $env:TEMP + "\ssm"
 
New-Item -ItemType directory -Path $dir -Force
cd $dir
(New-Object System.Net.WebClient).DownloadFile("https://amazon-ssm-$region.s3.$region.amazonaws.com/latest/windows_amd64/AmazonSSMAgentSetup.exe", $dir + "\AmazonSSMAgentSetup.exe")
Start-Process .\AmazonSSMAgentSetup.exe -ArgumentList @("/q", "/log", "install.log", "CODE=$code", "ID=$id", "REGION=$region") -Wait
Get-Content ($env:ProgramData + "\Amazon\SSM\InstanceData\registration")
Get-Service -Name "AmazonSSMAgent"

Note: that you should replace the URL in the example with the URL from your CloudFormation Stack Output.

Cleaning up

To clean up the environment, deregister the servers from Systems Manager. Then, delete the AWS CloudFormation stack that you created in the walkthrough by deleting Create API Gateway and Lambda CloudFormation Stack first followed by Create VPC endpoint for API Gateway CloudFormation Stack. At last, delete the KMS key created in the walkthrough.

Conclusion

In this post, I demonstrated how to automate Systems Manager Hybrid Activations creation. By adopting this solution, you can quickly register your hybrid environment devices to Systems Manager and minimize the overhead of managing the Hybrid Activations.

Author: