Building a Cross-Region/Cross-Account Code Deployment Solution on AWS

Many of our customers have expressed a desire to build an end-to-end release automation workflow solution that can deploy changes across multiple regions or different AWS accounts.

In this post, I will show you how you can easily build an automated cross-region code deployment solution using AWS CodePipeline (a continuous delivery service), AWS CodeDeploy (an automated application deployment service), and AWS Lambda (a serverless compute service). In the Taking This Further section, I will also show you how to extend what you’ve learned so that you can create a cross-account deployment solution.

We will use AWS CodeDeploy and AWS CodePipeline to create a multi-pipeline solution running in two regions (Region A and Region B). Any update to the source code in Region A will trigger validation and deployment of source code changes in the pipeline in Region A. A successful processing of source code in all of its AWS CodePipeline stages will invoke a Lambda function as a custom action, which will copy the source code into an S3 bucket in Region B. After the source code is copied into this bucket, it will trigger a similar chain of processes into the different AWS CodePipeline stages in Region B. See the following diagram.

This architecture follows best practices for multi-region deployments, sequentially deploying code into one region at a time upon successful testing and validation. This architecture lets you place controls to stop the deployment if a problem is identified with release. This prevents a bad version from being propagated to your next environments.

This post is based on the Simple Pipeline Walkthrough in the AWS CodePipeline User Guide. I have provided an AWS CloudFormation template that automates the steps for you.

Prerequisites

You will need an AWS account with administrator permissions. If you don’t have an account, you can sign up for one here. You will also need sample application source code that you can download here.

We will use the CloudFormation template provided in this post to create the following resources:

• Amazon S3 buckets to host the source code for the sample application. You can use a GitHub repository if you prefer, but you will need to change the CloudFormation template.
• AWS CodeDeploy to deploy the sample application.
• AWS CodePipeline with predefined stages for this setup.
• AWS Lambda as a custom action in AWS CodePipeline. It invokes a function to copy the source code into another region or account. If you are deploying to multiple accounts, cross-account S3 bucket permissions are required.

Note: The resources created by the CloudFormation template may result in charges to your account. The cost will depend on how long you keep the CloudFormation stack and its resources.

Let’s Get Started

Choose your source and destination regions for a continuous delivery of your source code. In this post, we are deploying the source code to two regions: first to Region A (Oregon) and then to Region B (N. Virginia/US Standard). You can choose to extend the setup to three or more regions if your business needs require it.

Step 1: Create Amazon S3 buckets for hosting your application source code in your source and destination regions. Make sure versioning is enabled on these buckets. For more information, see these steps in the AWS CodePipeline User Guide.

For example:

xrdeployment-sourcecode-us-west-2-<AccountID>           (Source code bucket in Region A – Oregon)

xrdeployment-sourcecode-us-east-1-<AccountID>           (Source code bucket in Region B – N. Virginia/US Standard)

Note: The source code bucket in Region B is also the destination bucket in Region A. Versioning on the bucket ensures that AWS CodePipeline is executed automatically when source code is changed.

Configuration Setup in Source Region A

Be sure you are in the US West (Oregon) region. You can use the drop-down menu to switch regions.

Step 2: In the AWS CloudFormation console, choose to launch the CloudFormation template. All of the steps in the Simple Pipeline Walkthrough are automated when you use this template.

This template creates a custom Lambda function and AWS CodePipeline and AWS CodeDeploy resources to deploy a sample application. You can customize any of these components according to your requirements later.

On the Specify Details page, do the following:

1. In Stack name, type a name for the stack (for example, XRDepDemoStackA).
2. In AppName, you can leave the default, or you can type a name of up to 40 characters. Use only lowercase letters, numbers, periods, and hyphens.
3. In InstanceCount and InstanceType, leave the default values. You might want to change them when you extend this setup for your use case.
4. In S3SourceCodeBucket, specify the name of the S3 bucket where source code is placed (xrdeployment-sourcecode-us-west-2-<AccountID>). See step 1.
5. In S3SourceCodeObject, specify the name of the source code zip file. The sample source code, xrcodedeploy_linux.zip, is provided for you.
6. Choose a destination region from the drop-down list. For the steps in this blog post, choose us-east-1.
7. In DestinationBucket, type the name of the bucket in the destination region where the source code will be copied  (xrdeployment-sourcecode-us-east-1-<AccountID>). See step 1.
8. In KeyPairName, choose the name of Amazon EC2 key pair. This enables remote login to your instances. You cannot sign in to your instance without generating a key pair and downloading a private key file. For information about generating a key pair, see these steps.
9. In SSHLocation, type the IP address from which you will access the resources created in this stack. This is a recommended security best practice.
10. In TagValue, type a value that identifies the deployment stage in the target deployment (for example, Alpha).
11. Choose Next.

(Optional) On the Options page, in Key, type Name. In Value, type a name that will help you easily identify the resources created in this stack. This name will be used to tag all of the resources created by the template. These tags are helpful if you want to use or modify these resources later on. Choose Next.

On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box. (It will.) Review the other settings, and then choose Create.

It will take several minutes for CloudFormation to create the resources on your behalf. You can watch the progress messages on the Events tab in the console.

When the stack has been created, you will see a CREATE_COMPLETE message in the Status column on the Overview tab.

Configuration Setup in Destination Region B

Step 3: We now need to create AWS resources in Region B. Use the drop-down menu to switch to US East (N. Virginia).

In the AWS CloudFormation console, choose  to launch the CloudFormation template.

On the Specify Details page, do the following:

1. In Stack name, type a name for the stack (for example, XRDepDemoStackB).
2. In AppName, you can leave the default, or you can type a name of up to 40 characters. Use only lowercase letters, numbers, periods, and hyphens.
3. In InstanceCount and InstanceType, leave the default values. You might want to change them when you extend this setup for your use case.
4. In S3SourceCodeBucket, specify the name of the S3 bucket where the source code is placed (xrdeployment-sourcecode-us-east-1-<AccountID>).  This is same as the DestinationBucket in step 2.
5. In S3SourceCodeObject, specify the name of the source code zip file. The sample source code (xrcodedeploy_linux.zip) is provided for you.
6. From the DestinationRegion drop-down list, choose none.
7. In DestinationBucket, type none. This is our final destination region for this setup.
8. In the KeyPairName, choose the name of the EC2 key pair.
9. In SSHLocation, type the IP address from which you will access the resources created in this stack.
10. In TagValue, type a value that identifies the deployment stage in the target deployment (for example, Beta).

Repeat the steps in the Configuration Setup in Source Region A until the CloudFormation stack has been created. You will see a CREATE_COMPLETE message in the Status column of the console.

So What Just Happened?

We have created an EC2 instance in both regions. These instances are running a sample web application. We have also configured AWS CodeDeploy deployment groups and created a pipeline where source changes propagate to AWS CodeDeploy groups in both regions. AWS CodeDeploy deploys a web page to each of the Amazon EC2 instances in the deployment groups. See the diagram at the beginning of this post.

The pipelines in both regions will start automatically as they are created. You can view your pipelines in the AWS CodePipeline console. You’ll find a link to AWS CodePipeline on the Outputs section of your CloudFormation stack.

Note: Your pipeline will fail during its first automatic execution because we haven’t placed source code into the S3SourceCodeBucket in the source region (Region A).

Step 4: Download the sample source code file, xrcodedeploy_linux.zip, from this link and place it in the source code S3 bucket for Region A. This will kick off AWS CodePipeline.

Step 5: Watch the progress of your pipeline in the source region (Region A) as it completes the actions configured for each of its stages and invokes a custom Lambda action that copies the source code into Region B. Then watch the progress of your pipeline in Region B (final destination region) after the pipeline succeeds in the source region (Region A). The pipeline in the destination region (Region B) should kick off automatically as soon as AWS CodePipeline in the source region (Region A) completes execution.

When each stage is complete, it turns from blue (in progress) to green (success).

Congratulations! You just created a cross-region deployment solution using AWS CodePipeline, AWS CodeDeploy, and AWS Lambda. You can place a new version of source code in your S3 bucket and watch it progress through AWS CodePipeline in all the regions.

Step 6: Verify your deployment. When Succeeded is displayed for the pipeline status in the final destination region, view the deployed application:

1. In the status area for Beta–stage in the final destination region, choose Details. The details of the deployment will appear in the AWS CodeDeploy console. You can also pick any other stage in other regions.
2. In the Deployment Details section, in Instance ID, choose the instance ID of any of the successfully deployed instance.
3. In the Amazon EC2 console, on the Description tab, in Public DNS, copy the address, and then paste it into the address bar of your web browser. The web page opens the sample web application that was built for you

Taking This Further

1. Using the CloudFormation template provided to you in this post, you can extend the setup to three regions.
2. So far we have deployed code in two regions within one AWS account. There may be a case where your environments exist in different AWS accounts. For example, assume a scenario in which:

• You have your development environment running in Region A in AWS Account A.
• You have your QA environment running in Region B in AWS Account B.
• You have a staging or production environment running in Region C in AWS Account C.

You will need to configure cross-account permissions on your destination S3 bucket and delegate these permissions to a role that Lambda assumed in the source account. Without these permissions, the Lambda function in AWS CodePipeline will not be able to copy the source code into the destination S3 bucket. (See the lambdaS3CopyRole in the CloudFormation template provided with this post.)

Create the following bucket policy on the destination bucket in Account B:

{

                  “Version”: “2012-10-17”,

                  “Statement”: [

                                    {

                                                      “Sid”: “DelegateS3Access”,

                                                      “Effect”: “Allow”,

                                                      “Principal”: {

                                                                        “AWS”: “arn:aws:iam::<Account A ID>:root”

                                                      },

                                                      “Action”: “s3:*”,

                                                      “Resource”: [

                                                                        “arn:aws:s3::: <destination bucket > /*”,

                                                                        “arn:aws:s3::: <destination bucket > “

                                                      ]

                                    }

                  ]

}

Repeat this step as you extend the setup to additional accounts.

Create your CloudFormation stacks in Account B and Account C (follow steps 2 and 3 in these accounts, respectively) and your pipeline will execute sequentially.

You can use another code repository solution like AWS CodeCommit or Github as your source and target repositories.

Wrapping Up

After you’ve finished exploring your pipeline and its associated resources, you can do the following:

• Extend the setup. Add more stages to your pipeline in AWS CodePipeline.
• Delete the stack in AWS CloudFormation, which deletes the pipeline, its resources, and the stack itself.

This is the option to choose if you no longer want to use the pipeline or any of its resources. Cleaning up resources you’re no longer using is important because you don’t want to continue to be charged.

To delete the CloudFormation stack:

1. Delete the Amazon S3 buckets used as the artifact store in AWS CodePipeline in the source and destination regions. Although this bucket was created as part of the CloudFormation stack, Amazon S3 does not allow CloudFormation to delete buckets that contain objects.To delete this bucket, open the Amazon S3 console, select the buckets you created in this setup, and then delete them. For more information, see Delete or Empty a Bucket.
2. Follow the steps to delete a stack in the AWS CloudFormation User Guide.

I would like to thank my colleagues Raul Frias, Asif Khan and Frank Li for their contributions to this post.