Creating a secure DevOps pipeline for AWS Service Catalog

Many AWS customers I speak with want to manage their AWS services using infrastructure as code (IaC) and DevOps practices for managing, versioning, and deploying products and portfolios. A best practice is deploying infrastructure templates from a continuous delivery (CD) pipeline with validation. In so doing, you can provide the AWS services your users need while enforcing security and governance best practices. This is especially relevant for companies in regulated industries that must implement segregation of duties, asset tracking, and audit trails.  Much like developers manage the vast number of files and dependencies with automation and version control, infrastructure engineers are also finding these CD pipelines critical to long-term success with increasingly large and complex cloud infrastructure.

In this post, I show you how to automate the deployment and validation of AWS CloudFormation templates using Stellingent’s CFN_NAG and AWS TaskCat. I also show you how to allow end user self-service while also enforcing least privilege provisioning using AWS Service Catalog and AWS Identity and Access Management (AWS IAM).

By following these directions, you can quickly deploy AWS CloudFormation templates through a CD pipeline from an AWS CodeCommit repository. These templates will be validated in the pipeline and then provided to end users through products and portfolios in AWS Service Catalog across multiple accounts and Regions. For this tutorial, I use git commands along with AWS Command Line Interface (AWS CLI) commands in a Linux terminal. Follow your organization’s standard process for configuring AWS CLI access to your AWS accounts. For more information on configuring the AWS CLI, see this documentation.

Overview

In order to fully realize the benefits of DevOps and IaC, it is important to organize your templates into separate files, which represent distinct and self-contained infrastructure tasks for your end users. This lets you to make smaller changes more often within specific areas of your infrastructure deployment. A best practice is to organize the infrastructure template files into three levels: portfolios, products, and resources. The portfolio templates represent a collection of products for a group of users. This means portfolios link the IAM roles, groups, users, tags, and actions that enable your end users to provision and manage those products. The product templates build contain the metadata that describe the products to your end users, such as support links, product names, and descriptions. The resource level represents the actual AWS services that are ultimately provisioned by your end users, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Virtual Private Cloud (Amazon VPC).

The CD pipeline uses AWS CodePipeline to manage the flow of three stages to deployment.

1. The automated process begins when an infrastructure engineer checks in the IaC template files to AWS CodeCommit.
2. The second stage contains two validation steps: cfn_nag and TaskCat
3. The last stage updates the CloudFormation StackSets. StackSets control the AWS Service Catalog products provided to your end users.

All the files used in this tutorial are located in the AWS Service Catalog reference architecture GitHub repository here: https://github.com/aws-samples/aws-service-catalog-reference-architectures.

While this pipeline can be used for a single account and Region, for this example I focus on multi-account, multi-Region deployments. For deployment through CloudFormation StackSets, the released AWS CloudFormation templates are stored in a single S3 bucket in the hub account. This S3 bucket provides your organization with a single, secured location for deploying approved and tested templates to all your accounts and Regions.

For an in-depth guide to deploying AWS Service Catalog across multiple accounts and Regions with StackSets, see this How to set up a multi-region, multi-account catalog of company standard AWS Service Catalog products blog post.

Setting up the hub and spoke accounts

The sharing of portfolios from hub to spoke can be simplified through AWS Organizations sharing features in AWS Service Catalog by following the directions in this blog. The instructions that follow are the manual steps required if you are not using AWS Organizations.

Choose an account to be your hub account. The hub account contains the AWS CodeCommit repository, deployed templates S3 bucket, and administrator role for AWS CloudFormation StackSets.

Write down your hub account ID and all spoke account IDs. To determine the account ID, use the following command in AWS CLI for each of your accounts. Your account ID is the “Account” value from the following CLI call:

aws sts get-caller-identity

Create the execution role in each spoke account

Use AWS CloudFormation StackSets to deploy AWS CloudFormation templates from your hub account to multiple spoke accounts. To do this, you must first create the execution IAM role in each spoke account. See the AWS CloudFormation User Guide for more details on setting up the required permissions.

• Choose a single Region, such as us-east-1, for deploying IAM roles. IAM roles are global and cover all Regions for a given account and should only be deployed once in a single Region.
• The hub account ID is a required parameter to create the trust relationship in each spoke account back to the hub. This trust allows the hub to execute AWS CloudFormation in the spoke.
• Log into each spoke account and execute the following CloudFormation Stack by clicking this Launchstack button or the link following it:
  
  
  (https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetExecutionRole.yml)

Execute the install script from the hub account

1. To get everything you need at once, clone the reference architecture repository in a bash terminal with the following command: git clone git@github.com:aws-samples/aws-service-catalog-reference-architectures.git
2. Go into the new directory and edit the codepipeline/install.sh file.  You must edit the line childAcc="" to include all of your spoke account IDs with a space between each number. If you have two spoke accounts, edit the line to include them with a single space between each account, as follows:  childAcc="123456789012 098765432100"  Do NOT include the hub account ID here.
3. The script is now ready to run. Ensure your AWS CLI is using the correct hub profile and Region then run the script from the AWS CodePipeline directory in a bash terminal:
   ./install.sh

It may take a few minutes for the script to finish. You may follow its progress in the AWS CloudFormation console. There is also a step-by-step guide for completing the setup without the script in the AWS CodePipeline section of the Github repository.

Once the script finishes, the initial creation of your AWS Service Catalog portfolio and products is complete. Your AWS CodePipeline project and CodeCommit repository are also complete.

Using the CD pipeline

The pipeline has been created, but the repository is empty.  Copy the deployed templates from the reference architecture repository to the new repository in your hub account. When you copy the folders from the reference architecture repository to your new AWS CodeCommit repository folder, do not copy the .git folder.

From a bash terminal, follow these commands to clone and populate your new AWS CodeCommit repository.

• To find the repository URL in the AWS Management Console of your hub account, look in the AWS CodeCommit console or the CloneUrlSsh output variable from the SC-RA-IACPipeline AWS CloudFormation stack.
• The SSH or HTTPS credential creation requires a manual one-time setup in IAM described in the documentation. If you do not have an IAM user with AWS CodeCommit permissions, then use this template iam/sc-codecommit-iamuser.yml to create one.
• Alternatively, you may use AWS Cloud9 as a terminal to execute the following commands without creating AWS CodeCommit credentials. A sample Cloud9 template can be found here https://github.com/aws-samples/aws-service-catalog-reference-architectures/blob/master/codepipeline/sc-cloud9.json

git clone YOUR-REPO-URL-HERE
rsync -av --exclude=.git aws-service-catalog-reference-architectures/ SCPortfoliosRepo/
cd SCPortfoliosRepo

A sample Cloud9 template can be found here https://github.com/aws-samples/aws-service-catalog-reference-architectures/blob/master/codepipeline/sc-cloud9.json.

Edit a product

The copied templates in your new repository are identical to the templates already deployed by the install script. Therefore, you must change a portfolio or product template in order for AWS CloudFormation to update. For example, in the ec2/sc-product-ec2-nginx-demowebserver.json file, remove one of the ProvisioningArtifactParameters to see the change reflected in AWS Service Catalog.

For AWS CloudFormation to update an AWS Service Catalog product, you must change both the resource file and the corresponding AWS Service Catalog product template. When changing AWS Service Catalog products, it is a best practice to add a new resource file and a new ProvisioningArtifact with a new product version in the product template when you change the underlying resource files.  Making new files for each new resource version will make it easier to manage and track multiple versions of infrastructure as it changes across long periods of time.

Watch the automation

Once you have made a change, you must commit and push the files to your remote CodeCommit repository in a bash terminal:

git add –all
git commit –m "first commit"
git push

AWS CodePipeline will automatically recognize the commit and proceed through the stages and actions. When the TaskCat validation and cfn_nag actions are successful, the deploy stage will update the AWS Service Catalog products.

You now have a working automated CD pipeline for updating an AWS Service Catalog portfolio. You can follow along with the automation and see any errors or validation failures in the AWS CloudFormation and AWS CodePipeline consoles.

Your automated pipeline for managing AWS Service Catalog is now set up and responding to template changes via git commits.

Enable end users to provision products

To allow your end users to provision products, add them to the ServiceCatalogEndusers IAM group. See the documentation for more information on granting user access to AWS Service Catalog.

Keep checking the reference architecture repository for updates and more examples as we are releasing new content to this site.

Conclusion

Customers combining DevOps practices with IaC have reduced their time managing infrastructure. They also are able to provide changes and new products to end users faster. By sharing AWS CloudFormation templates in a common repository teams can standardize on common templates which further increase inter-team collaboration and time to market for products with common infrastructure needs.

In this post, I showed you how to automate the deployment and validation of AWS CloudFormation templates using Stellingent’s CFN_NAG and AWS TaskCat.  This CD pipeline provides the infrastructure which allows end user self-service while also enforcing least privilege provisioning using AWS Service Catalog and IAM.