Packaging to Distribution – Using AWS Systems Manager Distributor to deploy Datadog

AWS Systems Manager Distributor automates the process of packaging and publishing software to managed Windows and Linux instances across the cloud landscape, as well as to on-premises servers, through a single simplified interface. Customers can now leverage AWS Systems Manager Distributor to package custom software like monitoring agents and security agents, and then distribute them to a large group of instances on a schedule or on-demand.

In this blog post, I’ll describe how to use Systems Manager Distributor to package and publish a monitoring agent, Datadog, across multiple AWS instances. As an example, I’ll describe the distribution of the Datadog agent version 6 to an AWS landscape that consists of Windows and Amazon Linux instances. Datadog is a SaaS-based monitoring and analytics platform for large-scale applications and infrastructure that delivers actionable alerts and powerful visualizations to provide full-stack observability.

The typical process for distributing the software packages to your AWS instances using Systems Manager Distributor involves the following steps.

Step 1: Create a package in Distributor

These are the steps for creating a package in Distributor:

Prepare the compressed zip folders of the Datadog agent
Upload the zip files to an Amazon S3 bucket
Prepare the manifest file
Create the package in the AWS Systems Manager Distributor console

Prepare the compressed zip folder of the Datadog agent

The Distributor package for any software includes two files apart from the actual software binaries. These files are install.ps1 and the uninstall.ps1 for Windows and install.sh and uninstall.sh for Linux. The installation commands for the software are included in the install.ps1/sh files, and the uninstallation commands are included in the uninstall.ps1/sh files.

The Datadog Agent webpage offers instructions on how to install their agent on many popular operating systems and distributions. I’ll describe how to create the Datadog Distributor packages for both Windows and Amazon Linux instances.

Windows instances

To target Windows instances, download the Datadog msi installation file from the URL provided on the Datadog Account settings page. Create a zip file with the installation file and two additional files: install.ps1 and uninstall.ps1. Let’s call this datadog-windows_6.zip.

The install.ps1 file includes the steps you must use deploy the Datadog agent in a silent fashion on your Windows instances using the command line.

You can reference the content of install.ps1 to deploy the Datadog agent as shown in the following code.

In this example, I’ll include the Datadog installation API key directly in the install script. Whenever you’re distributing potentially sensitive data like a key, make sure that you’re evaluating the impact and considering the use of other tools, like AWS Systems Manager Parameter Store and AWS Secrets Manager to secure access where necessary.

msiexec /qn /i datadog-agent-6-latest.amd64.msi APIKEY="<DATADOG APIKEY>" TAGS="Windows"

Based on your customization of the Datadog agent, you might need to include additional parameters in the install.ps1 file.

The uninstall.ps1 file includes the following reference instructions to uninstall the agent.

(Get-WmiObject -Class Win32_Product -Filter "Name='Datadog Agent'").Uninstall()

As an additional security measure to protect the integrity of the files, the checksum value of the zip file is verified during the package installation process. The checksum value is captured in the package manifest file during the package creation. After you create the zip file with the Datadog binaries, install.ps1, and uninstall.ps1 files, you can calculate the SHA256 checksum of the file Datadog-windows_6.zip. On a Windows system, you can use the following PowerShell command.

Get-FileHash -Algorithm SHA256 datadog-windows_6.zip

Linux instances

To target Amazon Linux instances, create a zip file with the Datadog installation file and two additional files: install.sh and uninstall.sh. Download the latest stable Datadog binaries from the yum repository URL provided at the Datadog Agent Upgrade page. Create a zip file with the binaries and the install.sh and uninstall.sh files. Let’s call this datadog-linux_6.zip.

Note: Although the binaries can be pulled during runtime from the Datadog yum repository without downloading the binaries beforehand, we recommend that you create the Distributor installation package zip file with the binaries and all required dependencies. This ensures that the package installation, including the dependent files, is consistent and has the same version across all instances. This makes it easier to maintain and troubleshoot.

A reference install.sh to deploy the Datadog agent is as below

yum install datadog-agent-6.9.0-1.x86_64.rpm --assumeyes
sed 's/^[:space:]*api_key:.*/api_key: <APIKEY>/' /etc/datadog-agent/datadog.yaml.example > /etc/datadog-agent/datadog.yaml
sudo systemctl start datadog-agent

The uninstall.sh file includes the instructions to uninstall the agent

yum remove datadog-agent --assumeyes

On Linux, you can use the below command to obtain the checksum

sha256sum datadog-linux_6.zip

Upload the zip files to an Amazon S3 bucket

Upload the datadog-windows.zip and datadog-linux.zip files to the S3 bucket created for the Distributor package. Configure the permissions on the bucket to have List and Get objects access the user/role that is creating the package in the Distributor.

Prepare the manifest file

A reference manifest file for the Datadog agent package looks like the following:

{
    "schemaVersion": "2.0",
    "version": "6.0.0",
    "packages": {
        "windows": {
            "_any": {
                "x86_64": {
                    "file": "datadog-windows_6.zip"
                }
            }
        },
        "amazon": {
            "_any": {
                "x86_64": {
                    "file": "datadog-linux_6.zip"
                }
            }
        }
    },
    "files": {
        "datadog-windows_6.zip": {
            "checksums": {
                "sha256": "<SHA256 value of the file datadog-windows_6.zip>"
            }
        },
        "datadog-linux_6.zip": {
            "checksums": {
                "sha256": "<SHA256 value of the file datadog-linux_6.zip>"
            }
        }
    }
}

Create the package in AWS Systems Manager console

Navigate to the Distributor page in AWS Systems Manager console and create the package. Provide the name of the package and a version associated with the package.

Enter the S3 bucket url where you uploaded the package earlier.

Choose New Manifest in the Manifest section and paste the contents of the manifest file that you prepared earlier.

Choose Create package button to create the Datadog package after completing all the above mentioned steps. Upon successful creation of the package, the package gets listed on the Distributor main page.

Step 2: Set up appropriate permissions on the package

By default, all packages are set to Private, meaning only those with access to the package creator’s AWS account can view package information, update or delete the package and distribute it.

Customers can distribute the package to all the systems managed by the Systems Manager using the Distributor. However, to distribute the package to systems in other AWS accounts managed by the systems manager of that account in the same Region, you can share the Datadog agent package by providing the account number of the target account in the Edit package permission box.

After the package is shared, you can access the package in the target accounts. In the target account, select the Region where the package was originally created at source, navigate to the Systems Manager Distributor page, and select the search filter as Account owner : Equal : All.

Step 3: Distribute the package

The following preparation steps are required before you can distribute the package to the target instances.

Ensure that the version of the Systems Manager agent running on the instances is 2.3.274.0 or later. To ensure the latest version of the Systems Manager agent is installed, run the AWS-UpdateSSMAgent document on all the targeted instances.

After your readiness is complete, you are ready to distribute the Datadog package. In the Systems Manager console, navigate to the datadog-agent package in the Distributor and select either Install one time or Install on schedule.

Install one time

When you choose Install one time, this takes you through the Run Command page of the Systems Manager console. In the command parameters, the Install action is selected by default for the ‘datadog-agent’. In the Version field, by default, the latest published version of the package is picked up for installation when the version field is left blank. Alternately, you can choose a specific version of the package to be installed as the following screenshot shows.

For Targets, select the instances using a tag. You can also manually select the instances. Then choose Run.

This will start distributing the package to the target instances.

Install on schedule

When you choose Install on schedule, the State Manager Create Association page opens. Provide an appropriate name for the association.

The Document section includes the ‘AWS-ConfigureAWSPackage’ document by default.

In Parameters, Action is set to the default, Install. Name defaults to the package name. In the Version field, by default, the latest published version of the package is picked up for installation when the version field is left blank. Alternately, you can choose a specific version of the package to be installed.

In the Targets section, choose the instances either through a tag or by manually selecting the instances. In the Specify schedule section, choose the schedule for running the association.

Optionally, in Advanced options, choose the Compliance severity, specify the Concurrency and Error Threshold in the Rate Control pane and Write to S3 in the Output Options window. Choose Create Association to kick-start the package distribution on schedule.

Step 4: Compliance Status

You can review the distribution status of the Datadog agent through the Run Command command history if the package is distributed using Install one time. You can review status in the State Manager association status if the package is distributed using the Install on schedule option.

If you used State Manager, you can also see the status of your package distribution through the Compliance console. The following screenshot shows the compliance summary.

Managing Package Versions

Managing changes and updates is part of the lifecycle of any software package. AWS Systems Manager Distributor provides the ability to manage this process.

If you want to make any changes to a package, like updating the version of the Datadog agent, adding support for a new operating system, or modifying the install or uninstall scripts, you can add a new version of the package. Once you’ve done that, both the old and new versions are available to install or uninstall on your instances.

To create a new version of the Datadog Distributor package, navigate to the datadog-agent package in the Distributor, and on the Versions tab, choose Add version.

This opens up the package version page with the Version name, S3 Bucket URL, and the Manifest file fields to capture the new package version specifics.

After you provide the details, choose Add version to add a new version to the package. The new version will appear in the Versions tab of the datadog-agent package.

Although all versions are available for you to distribute, you can mark one version as the default so that you can easily keep all of your instances up to date. To make a specific version the default, select the version of the package and choose the Set default version option. If you need to do a rollback, you’ll follow the same process of selecting a specific older version and distributing it, which will override the newer version on the managed instances.

Conclusion

In this blog post I covered the process of creating software packages and distributing them using AWS Systems Manager Distributor with Datadog agent as an example. I started by showing the overall process and steps involved. First, I created the Datadog software package through the Distributor. Then, I showed you how to configure the permissions on the package, distribute the package to the managed instances, and check the status of the distribution. Finally, I showed you how to create multiple versions of the package in line with software updates or changes to the installation commands.

About the author

Chandra Allaka is a Senior Consultant with AWS Professional Services. He is passionate about helping customers with their cloud production readiness and building effective solutions to manage cloud environments at scale.

AWS Cloud Operations Blog