AWS Marketplace

Extend and automate monitoring of multi-account AWS environments with Datadog and AWS Control Tower

As of December 2022, to integrate Datadog monitoring into your multi-account environment, we now recommend using Datadog’s Account Factory customization. To learn how to use Account Factory customization to deploy Datadog’s AWS Integration into new accounts, visit the most recent blog post.

Operational intelligence in a cloud environment helps organizations gain in-depth visibility into resources across multiple accounts and regions. Operational intelligence software solutions can help AWS Control Tower administrators automate tool integration, enabling complete visibility and insights in real time. Having a multi-account strategy is a best practice to achieve higher isolation of resources. It also helps to meet regulatory and compliance needs, track operational costs, and add an extra leyer of security. AWS Control Tower uses AWS best practices to establish a well-architected, multi-account baseline. It also enables governance across your AWS accounts.

Datadog is an AWS Advanced Technology Partner that helps you reduce risk during the modernization of infrastructure and applications. Datadog software enables you to immediately pull metrics from your AWS tools and services and view them. The integration also provides out-of-the-box  dashboards with visualizations for common metrics for popular AWS services such as Amazon EC2, Amazon S3, Amazon EMR, and Amazon DynamoDB. You can also create custom dashboards.

Many customers use AWS Control Tower to manage and govern multi-account AWS environments. For more information about managing multi account AWS environments with AWS Control Tower, see Getting Started with AWS Control Tower. 

In this post, we share a new solution that integrates AWS Control Tower with Datadog. The integration enables all newly added AWS accounts in an AWS Control Tower environment to be automatically enrolled with Datadog using Account Factory and AWS Control Tower Lifecycle Events. This provides full automation for AWS resource monitoring with Datadog in a multi-account AWS environment.

Prerequisites

You need the following prerequisite before implementing the Datadog and AWS Control Tower integration:

  • A Datadog subscription in AWS Marketplace. To get this, go to the AWS Marketplace listing for Datadog Pro and choose Continue to Subscribe. Choose the length of contract, 1 month or 12 months. In the contract options section, start the subscription by selecting 1 unit of Infra Hosts.

Solution overview

The AWS Control Tower integration with Datadog is based on automation of AWS Control Tower lifecycle events via AWS CloudWatch Events, AWS Lambda, and AWS CloudFormation StackSets. It consists of one AWS CloudFormation template that fully automates the provisioning, setup, and integration of all the components necessary for this solution.

Here is this AWS CloudFormation template. The template is deployed in the AWS Control Tower management account and it creates the following components:

A Datadog StackSet in the AWS Control Tower management account – Incorporates the Datadog components for setting up a Datadog forwarder and a Datadog integration role. All parameters needed for the Datadog components, such as the API key and secret, are stored in AWS Secrets Manager.

  • An Amazon CloudWatch Events rule – Triggered based on an AWS Control Tower lifecycle event.
  • An AWS Lambda lifecycle function – The target for the CloudWatch Events rule.

When a new account is added from the AWS Control Tower management account, the following occurs.

  1. The AWS Lambda lifecycle function invokes the Datadog AWS Integration API that registers the newly added AWS account in Datadog and generates a unique external ID in response.
  2. The Lambda function passes the external ID as a parameter into the Datadog stack instance and provisions it in the newly added managed account.

The following architecture illustrates the components of AWS Control Tower and the Datadog integration.

  • The AWS Control Tower management account consists of a CloudWatch Events rule that is triggered based on an AWS Control Tower lifecycle event.
  • The AWS Control Tower lifecycle event is itself triggered whenever a new AWS Control Tower account is provisioned. This account is provisioned via the Account Factory console that is part of AWS Service Catalog.
  • A Lambda lifecycle function is the target for the AWS CloudWatch Events rule.
  • The AWS Control Tower management account also consists of an AWS CloudFormation StackSet that is provisioned with the initial install of the AWS CloudFormation template. This AWS CloudFormation StackSet is the basis of the Datadog stack instances that are launched in the AWS Control Tower managed accounts when an AWS Control Tower lifecycle event is triggered. Refer to the following diagram.

Datadog and AWS Control Tower infrastructure diagram

  • The AWS Control Tower managed account consists of the Datadog stack instances that provision the Datadog forwarder Lambda function as well as the Datadog integration role.
  • The Datadog stack instances are provisioned in the AWS Control Tower managed account based on parameters initially supplied to the AWS CloudFormation StackSet that was provisioned in the AWS Control Tower management account. Refer to the following diagram.

Datadog and AWS Control Tower infrastructure diagram

Step-by-step walkthrough

Follow these steps to set up the Datadog integration with AWS Control Tower.

Set up Datadog integration with AWS Control Tower

  1. Log in to the Datadog console.
  2. Follow the steps outlined in the Datadog documentation to create an API key and an Application key in the Datadog console
  3. Log in to the AWS CloudFormation console of your management account.
  4. From the AWS CloudFormation console, launch the aws-datadog-controltower.yaml To launch a AWS CloudFormation template from the console, follow the steps outlined here.
    1. In the step to enter parameters from the steps outlined above, enter the API key and the Application Key. Accept all other default values for this template.

Test your integration

Test the integration by adding a managed account and creating a lifecycle event.

Add the managed account

  1. Log in to the AWS Control Tower management account and open the AWS Control Tower console.
  2. To enroll a new managed account in the AWS Control Tower organization, in the navigation pane, choose Account Factory.
  3. Enter values for Account email, Display name, AWS SSO email, AWS SSO user name, and Organizational unit.
  4. Choose Enroll account.

It can take up to 30 minutes for the account to be created and the AWS Control Tower lifecycle event to trigger.

Verify the integration works

To check that the integration is working, do the following:

  1. Log in to the AWS CloudFormation console of the managed account. In the left sidebar, choose Stacks.
    1. Check that the stacks for the DatadogForwarder and the Datadog Integration Role have been successfully provisioned. These stacks will have a name prefixed with StackSet-DataDogForwarderv1 and will show a status of CREATE_COMPLETE:.
  2. Navigate to the Datadog console. Locate the Amazon Web Services Integration tile and choose  Configurations. Verify the new managed account has been successfully registered. If the account was successfully registered, the Datadog console displays a message stating This integration is working properly and display the managed account’s 12-digit AWS Account ID. To ensure that the account has been successfully registered with Datadog, at the bottom left of the console, we recommend choosing Update Configuration.

Conclusion

In this blog post, we have demonstrated a solution that automatically enrolls new AWS Control Tower accounts with Datadog. Our solution for Datadog’s integration with AWS Control Tower now enables you to automatically extend the cloud monitoring and analytics capabilities of Datadog to your multi-account AWS environment. For more information about this solution, see Solutions for AWS Control Tower in AWS Marketplace.

About the authors

Kanishk Mahajan

Kanishk focuses on enabling ISV partners and our mutual customers with hands on solution architecture in their AWS cloud journey. This includes all aspects of AWS that relate to application modernization, secure cloud workload migration, infrastructure setup, and operations.

 

 

Raphael Sack

Raphael is a technical business development manager for AWS Service Catalog and AWS Control Tower. He enjoys tinkering with automation and code and active member of the management tools community.

 

 

 

Jimmy Caputo

Jimmy Caputo is a Senior Product Manager at Datadog. He leads a team of product managers responsible for Datadog’s integrations with AWS and other cloud providers, serverless monitoring, and IoT device monitoring.