Developing, versioning, testing, and deploying landing zone changes using CfCT across multiple landing zones

Enterprise customers often ask how they can minimize risk when they’re developing and testing a landing zone configuration. They also want to know how they can promote code between multiple landing zones.

­AWS Control Tower provides the easiest way to set up and govern a secure, multi-account AWS environment, called a landing zone. Customers who want to customize the AWS Control Tower landing zone can use the Customizations for AWS Control Tower (CfCT) solution. CfCT allows you to extend your Control Tower landing zone with a configuration package of infrastructure as code (IaC). The configuration package includes a manifest file, AWS CloudFormation templates, and JSON policy files. These files are packaged into a folder structure and placed in an Amazon Simple Storage Service (Amazon S3) bucket as a ZIP file.

When you’re developing a customization module for CfCT, we recommend to follow established DevOps practices for developing, versioning, testing, and deploying services. Many CfCT customizations deploy services that use organizational support, and because a CfCT landing zone manages account baselines and compliance at the AWS Organizations level, you need to test changes at an organizational level. If you develop and test customizations in an organization that contains production workloads, your work might affect those workloads and compliance baselines. During development and testing of CfCT customizations, it’s common to have AWS CodePipeline failures that cause delays to other necessary CfCT updates or prevent successful deployment within a maintenance window. For this reason, it’s crucial to develop, test, and deploy CfCT configuration code in other organizations before you deploy it in the production organization.

In this blog post, we share a method for developing, versioning, testing, and deploying landing zone changes using CfCT across multiple landing zones.

Reference architecture

AWS Control Tower environments

• In this scenario, the Development environment is an engineer-specific environment. In other words, for testing and educational purposes, each landing zone engineer has access to an AWS Control Tower development environment using the setup wizard from a test account. Development is a minimal environment implementation that contains only the organizational units (OUs) and accounts required to test the CfCT configurations.
• The PreProd environment acts as a staging environment to ensure that everything functions properly with the newly developed features and configurations. All deployment pipeline workflows and changes are verified here before the changes are pushed to Production. Like the Development environment, PreProd is a minimal environment implementation that contains only the OUs and accounts required to test the CfCT configurations.
• The Production environment is the complete Control Tower installation with all OUs and accounts required to support customer operations and application workloads.

Components

This reference architecture includes two categories of deployment pipelines:

• The CfCT Deploy pipelines are provided by the CfCT CloudFormation template and are provisioned in the Control Tower management account for each of the Control Tower environments (Development, PreProd, and Production). These pipelines deploy the Control Tower customizations, add-ons, and top-level cloud infrastructure service configurations. The CfCT Deploy pipelines are configured with an S3 bucket event trigger rather than an optional AWS CodeCommit branch trigger because we wanted to trigger these pipelines from a cross-account file drop event to avoid maintaining multiple Git remotes.
• The Transfer pipelines (one for each Control Tower environment) are located in the deployment account along with the CfCT CodeCommit repository. These pipelines are triggered through CfCT CodeCommit branch source pushes and serve three purposes:

1. 1. To perform customer-specific security checks, static code analysis, testing, and more outside of what’s already provided by the CfCT Customizations Using a second pipeline for customer-specific pipeline tasks allowed us to avoid making any modifications to the CfCT Deploy pipelines. This makes the upgrade path easier when the Control Tower team provides an update to the CfCT CloudFormation template. Customer-specific pipeline tasks vary by customer, but even more by technology stack. For simplicity, we’ll assume that all CfCT add-ons requiring an AWS Lambda function are developed with Python using the AWS SDK for Python (Boto3). For this configuration, we might run the following pipeline tasks to ensure the quality of our CloudFormation and Python Lambda function code:
      • cfn_nag: A CloudFormation linting utility that inspects AWS resource security configurations, and its capabilities are extensible through custom rules.
      • cfn_guard: An AWS open-source utility that allows developers to build custom enterprise policies to validate infrastructure against organizational best practices for security, compliance, and so on.
      • pylint and flake8: Python linters that enforce code syntax and style guide compliance.
      • pytest: A widely used Python stack unit test framework.
   2. To assemble the S3 deployment artifact. Because we’re using the CfCT CodeCommit repository as the source for all Control Tower environments, we needed to maintain a separate manifest.env.yml configuration for each Control Tower environment. We needed to ensure the correct environment manifest is used for the target environment and that it’s renamed manifest.yml, as expected by the CfCT Deploy pipelines.
   3. To perform a cross-account S3 artifact drop to trigger the CfCT Deploy pipeline in the target Control Tower environment.

An additional and optional purpose for the Transfer pipeline is to provide a manual approval action that serves as a breakpoint in the pipeline’s execution. Some organizations require a final approval stage before they deploy code changes. They might want to ensure changes happen during a release window or record an official approval event for infrastructure changes. As with the Python utilities discussed earlier, a manual approval is a customer-specific pipeline operation and belongs in the Transfer pipeline so no modifications need to be made to the CfCT Deploy pipeline.

Development workflow

Figure 1: Architecture diagram

1. A landing zone engineer creates a feature branch off of the develop branch in the CT Customizations repo and starts developing of a new infrastructure configuration or Control Tower add-on. Throughout the feature development iteration, the landing zone engineer deploys, as needed, to a Development Control Tower environment using an S3 deployment trigger (shown in the diagram) or a separate CodeCommit repo trigger.
2. When feature development is complete, the landing zone engineer creates a pull request from the feature branch to the develop branch.
3. When the pull request is approved, the merge operation triggers the PreProd Transfer pipeline.
   • Upon successful completion, the PreProd Transfer pipeline performs a cross-account S3 object drop for the PreProd CfCT Deploy pipeline.
   • The PreProd CfCT Deploy pipeline executes the deployment of the latest changes.

Steps 1-3 are repeated for as long as it takes to aggregate features and fixes for the work in the develop branch to be production-ready (perhaps for the duration of a development sprint).

4. When the develop branch is production-ready, the landing zone engineer creates a pull request for the develop branch to the main branch.
5. When the pull request is approved, the merge operation triggers the Production Transfer pipeline.
   • Upon successful completion, the Production Transfer pipeline performs a cross-account S3 object drop for the Production CfCT Deploy pipeline.
   • The Production CfCT Deploy pipeline executes the deployment of the latest changes.

Conclusion

In this post, we explained why you need multiple landing zones for developing and testing AWS Control Tower customizations. If you’re planning to build a brand-new landing zone or optimize an existing one on AWS, you can take advantage of this well-established architecture pattern to minimize the risk of disruption to the production workloads.

For more information, see Setting up a secure and scalable multi-account AWS environment in AWS Prescriptive Guidance.

About the authors