AWS DevOps Blog

Building a Cross-Region/Cross-Account Code Deployment Solution on AWS

by BK Chaurasiya | on | in How-to | | Comments

Many of our customers have expressed a desire to build an end-to-end release automation workflow solution that can deploy changes across multiple regions or different AWS accounts.

In this post, I will show you how you can easily build an automated cross-region code deployment solution using AWS CodePipeline (a continuous delivery service), AWS CodeDeploy (an automated application deployment service), and AWS Lambda (a serverless compute service). In the Taking This Further section, I will also show you how to extend what you’ve learned so that you can create a cross-account deployment solution.

We will use AWS CodeDeploy and AWS CodePipeline to create a multi-pipeline solution running in two regions (Region A and Region B). Any update to the source code in Region A will trigger validation and deployment of source code changes in the pipeline in Region A. A successful processing of source code in all of its AWS CodePipeline stages will invoke a Lambda function as a custom action, which will copy the source code into an S3 bucket in Region B. After the source code is copied into this bucket, it will trigger a similar chain of processes into the different AWS CodePipeline stages in Region B. See the following diagram.


This architecture follows best practices for multi-region deployments, sequentially deploying code into one region at a time upon successful testing and validation. This architecture lets you place controls to stop the deployment if a problem is identified with release. This prevents a bad version from being propagated to your next environments.

This post is based on the Simple Pipeline Walkthrough in the AWS CodePipeline User Guide. I have provided an AWS CloudFormation template that automates the steps for you.



You will need an AWS account with administrator permissions. If you don’t have an account, you can sign up for one here. You will also need sample application source code that you can download here.

We will use the CloudFormation template provided in this post to create the following resources:

  • Amazon S3 buckets to host the source code for the sample application. You can use a GitHub repository if you prefer, but you will need to change the CloudFormation template.
  • AWS CodeDeploy to deploy the sample application.
  • AWS CodePipeline with predefined stages for this setup.
  • AWS Lambda as a custom action in AWS CodePipeline. It invokes a function to copy the source code into another region or account. If you are deploying to multiple accounts, cross-account S3 bucket permissions are required.

Note: The resources created by the CloudFormation template may result in charges to your account. The cost will depend on how long you keep the CloudFormation stack and its resources.

Let’s Get Started

Choose your source and destination regions for a continuous delivery of your source code. In this post, we are deploying the source code to two regions: first to Region A (Oregon) and then to Region B (N. Virginia/US Standard). You can choose to extend the setup to three or more regions if your business needs require it.

Step 1: Create Amazon S3 buckets for hosting your application source code in your source and destination regions. Make sure versioning is enabled on these buckets. For more information, see these steps in the AWS CodePipeline User Guide.

For example:

xrdeployment-sourcecode-us-west-2-<AccountID>           (Source code bucket in Region A – Oregon)

xrdeployment-sourcecode-us-east-1-<AccountID>           (Source code bucket in Region B – N. Virginia/US Standard)

Note: The source code bucket in Region B is also the destination bucket in Region A. Versioning on the bucket ensures that AWS CodePipeline is executed automatically when source code is changed.

Configuration Setup in Source Region A

Be sure you are in the US West (Oregon) region. You can use the drop-down menu to switch regions.


Step 2: In the AWS CloudFormation console, choose launch-stack to launch the CloudFormation template. All of the steps in the Simple Pipeline Walkthrough are automated when you use this template.

This template creates a custom Lambda function and AWS CodePipeline and AWS CodeDeploy resources to deploy a sample application. You can customize any of these components according to your requirements later.

On the Specify Details page, do the following:

  1. In Stack name, type a name for the stack (for example, XRDepDemoStackA).
  2. In AppName, you can leave the default, or you can type a name of up to 40 characters. Use only lowercase letters, numbers, periods, and hyphens.
  3. In InstanceCount and InstanceType, leave the default values. You might want to change them when you extend this setup for your use case.
  4. In S3SourceCodeBucket, specify the name of the S3 bucket where source code is placed (xrdeployment-sourcecode-us-west-2-<AccountID>). See step 1.
  5. In S3SourceCodeObject, specify the name of the source code zip file. The sample source code,, is provided for you.
  6. Choose a destination region from the drop-down list. For the steps in this blog post, choose us-east-1.
  7. In DestinationBucket, type the name of the bucket in the destination region where the source code will be copied  (xrdeployment-sourcecode-us-east-1-<AccountID>). See step 1.
  8. In KeyPairName, choose the name of Amazon EC2 key pair. This enables remote login to your instances. You cannot sign in to your instance without generating a key pair and downloading a private key file. For information about generating a key pair, see these steps.
  9. In SSHLocation, type the IP address from which you will access the resources created in this stack. This is a recommended security best practice.
  10. In TagValue, type a value that identifies the deployment stage in the target deployment (for example, Alpha).
  11. Choose Next.



(Optional) On the Options page, in Key, type Name. In Value, type a name that will help you easily identify the resources created in this stack. This name will be used to tag all of the resources created by the template. These tags are helpful if you want to use or modify these resources later on. Choose Next.

On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box. (It will.) Review the other settings, and then choose Create.


It will take several minutes for CloudFormation to create the resources on your behalf. You can watch the progress messages on the Events tab in the console.

When the stack has been created, you will see a CREATE_COMPLETE message in the Status column on the Overview tab.


Configuration Setup in Destination Region B

Step 3: We now need to create AWS resources in Region B. Use the drop-down menu to switch to US East (N. Virginia).

In the AWS CloudFormation console, choose launch-stack to launch the CloudFormation template.

On the Specify Details page, do the following:

  1. In Stack name, type a name for the stack (for example, XRDepDemoStackB).
  2. In AppName, you can leave the default, or you can type a name of up to 40 characters. Use only lowercase letters, numbers, periods, and hyphens.
  3. In InstanceCount and InstanceType, leave the default values. You might want to change them when you extend this setup for your use case.
  4. In S3SourceCodeBucket, specify the name of the S3 bucket where the source code is placed (xrdeployment-sourcecode-us-east-1-<AccountID>).  This is same as the DestinationBucket in step 2.
  5. In S3SourceCodeObject, specify the name of the source code zip file. The sample source code ( is provided for you.
  6. From the DestinationRegion drop-down list, choose none.
  7. In DestinationBucket, type none. This is our final destination region for this setup.
  8. In the KeyPairName, choose the name of the EC2 key pair.
  9. In SSHLocation, type the IP address from which you will access the resources created in this stack.
  10. In TagValue, type a value that identifies the deployment stage in the target deployment (for example, Beta).

Repeat the steps in the Configuration Setup in Source Region A until the CloudFormation stack has been created. You will see a CREATE_COMPLETE message in the Status column of the console.

So What Just Happened?

We have created an EC2 instance in both regions. These instances are running a sample web application. We have also configured AWS CodeDeploy deployment groups and created a pipeline where source changes propagate to AWS CodeDeploy groups in both regions. AWS CodeDeploy deploys a web page to each of the Amazon EC2 instances in the deployment groups. See the diagram at the beginning of this post.

The pipelines in both regions will start automatically as they are created. You can view your pipelines in the AWS CodePipeline console. You’ll find a link to AWS CodePipeline on the Outputs section of your CloudFormation stack.


Note: Your pipeline will fail during its first automatic execution because we haven’t placed source code into the S3SourceCodeBucket in the source region (Region A).

Step 4: Download the sample source code file,, from this link and place it in the source code S3 bucket for Region A. This will kick off AWS CodePipeline.

Step 5: Watch the progress of your pipeline in the source region (Region A) as it completes the actions configured for each of its stages and invokes a custom Lambda action that copies the source code into Region B. Then watch the progress of your pipeline in Region B (final destination region) after the pipeline succeeds in the source region (Region A). The pipeline in the destination region (Region B) should kick off automatically as soon as AWS CodePipeline in the source region (Region A) completes execution.

When each stage is complete, it turns from blue (in progress) to green (success).


Congratulations! You just created a cross-region deployment solution using AWS CodePipeline, AWS CodeDeploy, and AWS Lambda. You can place a new version of source code in your S3 bucket and watch it progress through AWS CodePipeline in all the regions.

Step 6: Verify your deployment. When Succeeded is displayed for the pipeline status in the final destination region, view the deployed application:

  1. In the status area for Betastage in the final destination region, choose Details. The details of the deployment will appear in the AWS CodeDeploy console. You can also pick any other stage in other regions.
  2. In the Deployment Details section, in Instance ID, choose the instance ID of any of the successfully deployed instance.
  3. In the Amazon EC2 console, on the Description tab, in Public DNS, copy the address, and then paste it into the address bar of your web browser. The web page opens the sample web application that was built for you


Taking This Further

  1. Using the CloudFormation template provided to you in this post, you can extend the setup to three regions.
  2. So far we have deployed code in two regions within one AWS account. There may be a case where your environments exist in different AWS accounts. For example, assume a scenario in which:
  • You have your development environment running in Region A in AWS Account A.
  • You have your QA environment running in Region B in AWS Account B.
  • You have a staging or production environment running in Region C in AWS Account C.



You will need to configure cross-account permissions on your destination S3 bucket and delegate these permissions to a role that Lambda assumed in the source account. Without these permissions, the Lambda function in AWS CodePipeline will not be able to copy the source code into the destination S3 bucket. (See the lambdaS3CopyRole in the CloudFormation template provided with this post.)

Create the following bucket policy on the destination bucket in Account B:


                  “Version”: “2012-10-17”,

                  “Statement”: [


                                                      “Sid”: “DelegateS3Access”,

                                                      “Effect”: “Allow”,

                                                      “Principal”: {

                                                                        “AWS”: “arn:aws:iam::<Account A ID>:root”


                                                      “Action”: “s3:*”,

                                                      “Resource”: [

                                                                        “arn:aws:s3::: <destination bucket > /*”,

                                                                        “arn:aws:s3::: <destination bucket > “






Repeat this step as you extend the setup to additional accounts.

Create your CloudFormation stacks in Account B and Account C (follow steps 2 and 3 in these accounts, respectively) and your pipeline will execute sequentially.

You can use another code repository solution like AWS CodeCommit or Github as your source and target repositories.

Wrapping Up

After you’ve finished exploring your pipeline and its associated resources, you can do the following:

  • Extend the setup. Add more stages to your pipeline in AWS CodePipeline.
  • Delete the stack in AWS CloudFormation, which deletes the pipeline, its resources, and the stack itself.

This is the option to choose if you no longer want to use the pipeline or any of its resources. Cleaning up resources you’re no longer using is important because you don’t want to continue to be charged.

To delete the CloudFormation stack:

  1. Delete the Amazon S3 buckets used as the artifact store in AWS CodePipeline in the source and destination regions. Although this bucket was created as part of the CloudFormation stack, Amazon S3 does not allow CloudFormation to delete buckets that contain objects.To delete this bucket, open the Amazon S3 console, select the buckets you created in this setup, and then delete them. For more information, see Delete or Empty a Bucket.
  2. Follow the steps to delete a stack in the AWS CloudFormation User Guide.


I would like to thank my colleagues Raul Frias, Asif Khan and Frank Li for their contributions to this post.

From ELK Stack to EKK: Aggregating and Analyzing Apache Logs with Amazon Elasticsearch Service, Amazon Kinesis, and Kibana

by Pubali Sen | on | in How-to |

By Pubali Sen, Shankar Ramachandran

Log aggregation is critical to your operational infrastructure. A reliable, secure, and scalable log aggregation solution makes all the difference during a crunch-time debugging session.

In this post, we explore an alternative to the popular log aggregation solution, the ELK stack (Elasticsearch, Logstash, and Kibana): the EKK stack (Amazon Elasticsearch Service, Amazon Kinesis, and Kibana). The EKK solution eliminates the undifferentiated heavy lifting of deploying, managing, and scaling your log aggregation solution. With the EKK stack, you can focus on analyzing logs and debugging your application, instead of managing and scaling the system that aggregates the logs.

In this blog post, we describe how to use an EKK stack to monitor Apache logs. Let’s look at the components of the EKK solution.

Amazon Elasticsearch Service is a popular search and analytics engine that provides real-time application monitoring and log and clickstream analytics. For this post, you will store and index Apache logs in Amazon ES. As a managed service, Amazon ES is easy to deploy, operate, and scale in the AWS Cloud. Using a managed service also eliminates administrative overhead, like patch management, failure detection, node replacement, backing up, and monitoring. Because Amazon ES includes built-in integration with Kibana, it eliminates installing and configuring that platform. This simplifies your process further. For more information about Amazon ES, see the Amazon Elasticsearch Service detail page.

Amazon Kinesis Agent is an easy-to-install standalone Java software application that collects and sends data. The agent continuously monitors the Apache log file and ships new data to the delivery stream. This agent is also responsible for file rotation, checkpointing, retrying upon failures, and delivering the log data reliably and in a timely manner. For more information, see Writing to Amazon Kinesis Firehose Using Amazon Kinesis Agent or Amazon Kinesis Agent in GitHub.

Amazon Kinesis Firehose provides the easiest way to load streaming data into AWS. In this post, Firehose helps you capture and automatically load the streaming log data to Amazon ES and back it up in Amazon Simple Storage Service (Amazon S3). For more information, see the Amazon Kinesis Firehose detail page.

You’ll provision an EKK stack by using an AWS CloudFormation template. The template provisions an Apache web server and sends the Apache access logs to an Amazon ES cluster using Amazon Kinesis Agent and Firehose. You’ll back up the logs to an S3 bucket. To see the logs, you’ll leverage the Amazon ES Kibana endpoint.

By using the template, you can quickly complete the following tasks:

·      Provision an Amazon ES cluster.

·      Provision an Amazon Elastic Compute Cloud (Amazon EC2) instance.

·      Install Apache HTTP Server version 2.4.23.

·      Install the Amazon Kinesis Agent on the web server.

·      Provision an Elastic Load Balancing load balancer.

·      Create the Amazon ES index and the associated log mappings.

·      Create an Amazon Kinesis Firehose delivery stream.

·      Create all AWS Identity and Access Management (IAM) roles and policies. For example, the Firehose delivery stream backs up the Apache logs to an S3 bucket. This requires that the Firehose delivery stream be associated with a role that gives it permission to upload the logs to the correct S3 bucket.

·      Configure Amazon CloudWatch Logs log streams and log groups for the Firehose delivery stream. This helps you to troubleshoot when the log events don’t reach their destination.

EKK Stack Architecture
The following architecture diagram shows how an EKK stack works.


To build the EKK stack, you must have the following:

·      An Amazon EC2 key pair in the US West (Oregon) Region. If you don’t have one, create one.

·      An S3 bucket in the US West (Oregon) Region. If you don’t have one, create one.

·      A default VPC in the US West (Oregon) Region. If you have deleted the default VPC, request one.

·      Administrator-level permissions in IAM to enable Amazon ES and Amazon S3 to receive the log data from the EC2 instance through Firehose.

Getting Started
Begin by launching the AWS CloudFormation template to create the stack.

1.     In the AWS CloudFormation console, choose  to   launch-stack the AWS CloudFormation template. Make sure that you are in the US West (Oregon) region.

Note: If you want to download the template to your computer and then upload it to AWS CloudFormation, you can do so from this Amazon S3 bucket. Save the template to a location on your computer that’s easy to remember.

2.     Choose Next.

3.     On the Specify Details page, provide the following:

Screen Shot 2016-11-01 at 9.44.20 AM

a)    Stack Name: A name for your stack.

b)    InstanceType: Select the instance family for the EC2 instance hosting the web server.

c)     KeyName: Select the Amazon EC2 key pair in the US West (Oregon) Region.

d)    SSHLocation: The IP address range that can be used to connect to the EC2 instance by using SSH. Accept the default,

e)    WebserverPort: The TCP/IP port of the web server. Accept the default, 80.

4.     Choose Next.

5.     On the Options page, optionally specify tags for your AWS CloudFormation template, and then choose Next.


6.     On the Review page, review your template details. Select the Acknowledgement checkbox, and then choose Create to create the stack.

It takes about 10-15 minutes to create the entire stack.

Configure the Amazon Kinesis Agent
After AWS CloudFormation has created the stack, configure the Amazon Kinesis Agent.

1.     In the AWS CloudFormation console, choose the Resources tab to find the Firehose delivery stream name. You need this to configure the agent. Record this value because you will need it in step 3.


2.     On the Outputs tab, find and record the public IP address of the web server. You need it to connect to the web server using SSH to configure the agent. For instructions on how to connect to an EC2 instance using SSH, see Connecting to Your Linux Instance Using SSH.


3. On the web server’s command line, run the following command:

sudo vi /etc/aws-kinesis/agent.json

This command opens the configuration file, agent.json, as follows.

{ "cloudwatch.emitMetrics": true, "firehose.endpoint": "", "awsAccessKeyId": "", "awsSecretAccessKey": "", "flows": [ { "filePattern": "/var/log/httpd/access_log", "deliveryStream": "", "dataProcessingOptions": [ { "optionName": "LOGTOJSON", "logFormat": "COMMONAPACHELOG" } ] } ] } 

4.     For the deliveryStream key, type the value of the KinesisFirehoseDeliveryName that you retrieved from the stack’s Resources tab. After you type the value, save and terminate the agent.json file.

5.     Run the following command on the CLI:

sudo service aws-kinesis-agent restart

6.     On the AWS CloudFormation console choose the resources tab and note the name of the Amazon ES cluster corresponding to the LogicalID ESDomain.

7.     Go to AWS Management Console, and choose Amazon Elasticsearch Service. Under My Domains, you can see the Amazon ES domain that the AWS CloudFormation template created.


Configure Kibana and View Your Apache Logs
Amazon ES provides a default installation of Kibana with every Amazon ES domain. You can find the Kibana endpoint on your domain dashboard in the Amazon ES console.

1.     In the Amazon ES console, choose the Kibana endpoint.

2.     In Kibana, for Index name or pattern, type logmonitor. logmonitor is the name of the AWS ES index that you created for the web server access logs. The health checks from Amazon Elastic Load Balancing generate access logs on the web server, which flow through the EKK pipeline to Kibana for discovery and visualization.

3.     In Time-field name, select datetime.


4.     On the Kibana console, choose the Discover tab to see the Apache logs.


Use Kibana to visualize the log data by creating bar charts, line and scatter plots, histograms, pie charts, etc.


Pie chart of IP addresses accessing the web server in the last 30 days


Bar chart of IP addresses accessing the web server in the last 5 minutes

You can graph information about http response, bytes, or IP address to provide meaningful insights on the Apache logs. Kibana also facilitates making dashboards by combining graphs.

Monitor Your Log Aggregator

To monitor the Firehose delivery stream, navigate to the Firehose console. Choose the stream, and then choose the Monitoring tab to see the Amazon CloudWatch metrics for the stream.



When log delivery fails, the Amazon S3 and Amazon ES logs help you troubleshoot. For example, the following screenshot shows logs when delivery to an Amazon ES destination fails because the date mapping on the index was not in line with the ingest log.


In this post, we showed how to ship Apache logs to Kibana by using Amazon Kinesis Agent, Amazon ES, and Firehose. It’s worth pointing out that Firehose automatically scales up or down based on the rate at which your application generates logs. To learn more about scaling Amazon ES clusters, see the Amazon Elasticsearch Service Developer Guide.

Managed services like Amazon ES and Amazon Kinesis Firehose simplify provisioning and managing a log aggregation system. The ability to run SQL queries against your streaming log data using Amazon Kinesis Analytics further strengthens the case for using an EKK stack. The AWS CloudFormation template used in this post is available to extend and build your own EKK stack.


Building End-to-End Continuous Delivery and Deployment Pipelines in AWS and TeamCity

by Balaji Iyer | on | in Best practices, How-to, Partners, Web app | | Comments

By Balaji Iyer, Janisha Anand, and Frank Li

Organizations that transform their applications to cloud-optimized architectures need a seamless, end-to-end continuous delivery and deployment workflow: from source code, to build, to deployment, to software delivery.

Continuous delivery is a DevOps software development practice where code changes are automatically built, tested, and prepared for a release to production. The practice expands on continuous integration by deploying all code changes to a testing environment and/or a production environment after the build stage. When continuous delivery is implemented properly, developers will always have a deployment-ready build artifact that has undergone a standardized test process.

Continuous deployment is the process of deploying application revisions to a production environment automatically, without explicit approval from a developer. This process makes the entire software release process automated. Features are released as soon as they are ready, providing maximum value to customers.

These two techniques enable development teams to deploy software rapidly, repeatedly, and reliably.

In this post, we will build an end-to-end continuous deployment and delivery pipeline using AWS CodePipeline (a fully managed continuous delivery service), AWS CodeDeploy (an automated application deployment service), and TeamCity’s AWS CodePipeline plugin. We will use AWS CloudFormation to setup and configure the end-to-end infrastructure and application stacks. The ­­pipeline pulls source code from an Amazon S3 bucket, an AWS CodeCommit repository, or a GitHub repository. The source code will then be built and tested using TeamCity’s continuous integration server. Then AWS CodeDeploy will deploy the compiled and tested code to Amazon EC2 instances.


You’ll need an AWS account, an Amazon EC2 key pair, and administrator-level permissions for AWS Identity and Access Management (IAM), AWS CloudFormation, AWS CodeDeploy, AWS CodePipeline, Amazon EC2, and Amazon S3.


Here are the steps:

  1. Continuous integration server setup using TeamCity.
  2. Continuous deployment using AWS CodeDeploy.
  3. Building a delivery pipeline using AWS CodePipeline.

In less than an hour, you’ll have an end-to-end, fully-automated continuous integration, continuous deployment, and delivery pipeline for your application. Let’s get started!


AWS Config: Checking for Compliance with New Managed Rule Options

by Shawn O'Connor | on |

Change is the inherent nature of the Cloud. New ideas can immediately take shape through the provisioning of resources, projects will iterate and evolve, and sometimes we must go back to the drawing board. Amazon Web Services accelerates this process by providing the ability to programmatically provision and de-provision resources. However, this freedom requires some responsibility on the part the organization in implementing consistent business and security policies.

AWS Config enables AWS resource inventory and change management as well as Config Rules to confirm that resources are configured in compliance with policies that you define. New options are now available for AWS Config managed rules, providing additional flexibility with regards to the type of rules and policies that can be created.

  • AWS Config rules can now check that running instances are using approved Amazon Machine Images, or AMIs. You can specify a list of approved AMI by ID or provide a tag to specify the list of AMI Ids.
  • The Required-Tag managed rule can now require up to 6 tags with optional values in a single rule. Previously, each rule accepted only a single tag/value combo.
  • Additionally, the Required-Tag managed rule now accepts a comma-separated list of values for each checked tag. This allows for a rule to be compliant if any one of a supplied list of tags is present on the resource.

A Sample Policy

Let’s explore how these new rule options can be incorporated into a broader compliance policy. A typical company’s policy might state that:

  • All data must be encrypted at rest.
  • The AWS IAM password policy must meet the corporate standard.
  • Resources must be billed to the correct cost center.
  • We want to know who owns a resource in case there is an issue or question about the resource.
  • We want to identify whether a resource is a part of Dev, QA, Production, or staging so that we can apply the correct SLAs and make sure the appropriate teams have access.
  • We need to identify any systems that handled privileged information such as Personally Identifiable Information (PII) or credit card data (PCI) as a part of our compliance requirements.
  • Our Ops teams regularly provides hardened images with the latest patches and required software (e.g. security and/or configuration management agents). We want to ensure that all Linux instances are built with these images. We do regular reviews to ensure our images are up to date.

We see organizations implementing a variety of strategies like the one we have defined. Tagging is typically used to categorize and identify resources. We will be using tags and AWS Config managed rules to satisfy each of our policy requirements. In addition to the AWS Config managed rules, custom compliance rules can be developed using AWS Lambda functions.


In the AWS Management console, we have built 5 rules.

  • strong_password_policy – Checks that we have a Strong Password policy for AWS IAM Users
  • only_encrypted_volumes – Checks that all attached EBS volumes are encrypted
  • approved_ami_by_id – Checks that approved AMI IDs have been used
  • approved_ami_by_tag – Check that AMIs tagged as sec_approved have been used
  • ec2_required_tags – Check that all required tags have been applied to EC2 instances:
    Tag Name (Key) Tag Value(s) Description
    Environment Prod / Dev / QA / Staging The environment in which the resource deployed
    Owner Joe, Steve, Anne Who is responsible?
    CostCenter General / Ops / Partner / Marketing Who is paying for the resource?
    LastReviewed Date (e.g. 09272016) Last time this instance was reviewed for compliance
    AuditLevel Normal / PII / PCI The required audit level for the instance

Strong Password Policy

AWS Config has pre-built rules to check against the configured IAM password Policy. We have implemented this rule using the default configurations.

We can see that our IAM password policy is Compliant with our managed rule.


Approved AMIs

We have our hardened AMI from the Ops team. The image also includes encrypted EBS volumes. We have added a tag called sec_approved that we will use to test our image.


We will create two additional rules to check that EC2 instances are launched with our approved AMIs. One will check that our sec_approved tag is present on the AMI. A second rule checks that the AMI ID (ami-1480c503) is included in a comma-separated list of approved images. Now we will launch two instances. One with a standard Amazon Linux AMI (i-019b0e790a67dd7f1) and one instance with our approved AMI (i-0caf40fcb4f48517c).


If we look at the details of the approved_amis_by_tag and approved_amis_by_id rules, we see that the instance that was launched with the unapproved image (i-019b0e790a67dd7f1) has been marked Noncompliant. Our hardened instance (i-0caf40fcb4f48517c) is Compliant.



Encrypted EBS Volumes

In the EC2 Console we see the volumes attached to our instances. We can see that the EBS volumes created by our hardened AMI have been encrypted as configured. They are attached to instance i-0caf40fcb4f48517c.


If we look at the details of the only_encrypted_volumes rule, we see that the instance that was launched with the unapproved image (i-019b0e790a67dd7f1) has been marked Noncompliant. Our hardened instance is Compliant.


Required Tags

Finally, let’s take a look at the ec2_required_tags rule. We have defined our required tags in our AWS Config Rule. Specific tag values are optional, but will be required if specified. Our hardened instance was configured with all required tags.



We can see that our tagged instance is compliant with our Required Tags Rule as well.


OpsWorks September 2016 Updates

by Daniel Huesch | on | | Comments

Over the past few months, the AWS OpsWorks team has introduced several enhancements to existing features and added to support for new one. Let’s discuss some of these new capabilities.

·       Chef client 12.13.37 – Released a new AWS OpsWorks agent version for Chef 12 for Linux, enabling the latest enhancements from Chef. The OpsWorks console now shows the full history of enhancements to its agent software. Here’s an example of what the change log looks like:

·       Node.js 0.12.15 – Provided support for a new version of Node.js, in Chef 11.

–        Fixes a bug in the read/write locks implementation for the Windows operating system.
–        Fixes a potential buffer overflow vulnerability.

·       Ruby 2.3.1 – The built-in Chef 11 Ruby layer now supports Ruby 2.3.1, which includes these Ruby enhancements:

–        Introduced a frozen string literal pragma.
–        Introduced a safe navigation operator (lonely operator).
–        Numerous performance improvements.

·       Larger EBS volumes – Following the recent announcement from Amazon EBS, you can now use OpsWorks to create provisioned IOPS volumes that store up to 16 TB and process up to 20,000 IOPS, with a maximum throughput of 320 MBps. You can also create general purpose volumes that store up to 16 TB and process up to 10,000 IOPS, with a maximum throughput of 160 MBps.

·       New Linux operating systems – OpsWorks continues to enhance its operating system support and now offers:

–        Amazon Linux 2016.03 (Amazon Linux 2016.09 support will be available soon)
–        Ubuntu 16.04
–        CentOS 7

·       Instance tenancy – You can provision dedicated instances through OpsWorks. Dedicated instances are Amazon EC2 instances that run in a virtual private cloud (VPC) on hardware that’s dedicated to a single customer. Your dedicated instances are physically isolated at the host hardware level from instances that belong to other AWS accounts.

·       Define root volumes – You can define the size of the root volume of your EBS-backed instances directly from the OpsWorks console. Choose from a variety of volume types: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic.

·       Instance page – The OpsWorks instance page now displays a summary bar that indicates the aggregated state of all the instances in a selected stack. Summary fields include total instance count, online instances, instances that are in the setting-up stage, instances that are in the shutting-down stage, stopped instances, and instances in an error state.

·       Service role regeneration – You can now use the OpsWorks console to recreate your IAM service role if it was deleted.

Recreate IAM service role

Confirmation of IAM service role creation

As always, we welcome your feedback about features you’re using in OpsWorks. Be sure to visit the OpsWorks user forums, and check out the documentation.



Secure AWS CodeCommit with Multi-Factor Authentication

by Steffen Grunwald | on | in How-to | | Comments

This blog post shows you how to set up AWS CodeCommit if you want to enforce multi-factor authentication (MFA) for your repository users. One of the most common reasons for using MFA for your AWS CodeCommit repository is to secure sensitive data or prevent accidental pushes to the repository that could trigger a sensitive change process.

By using the MFA capabilities of AWS Identity and Access Management (IAM) you can add an extra layer of protection to sensitive code in your AWS CodeCommit repository. AWS Security Token Service (STS) and IAM allow you to stretch the period during which the authentication is valid from 15 minutes to 36 hours, depending on your needs. AWS CLI profile configuration and the AWS CodeCommit credential helper transparently use the MFA information as soon as it has been issued, so you can work with MFA with minimal impact to your daily development process.

Solution Overview

AWS CodeCommit currently provides two communication protocols and authentication methods:

  • SSH authentication uses keys configured in IAM user profiles.
  • HTTPS authentication uses IAM keys or temporary security credentials retrieved when assuming an IAM role.

It is possible to use SSH in a manner that incorporates multiple factors. An SSH private key can be considered something you have and its passphrase something you know. However, the passphrase cannot technically be enforced on the client side. Neither is it issued on an independent device.

That is why the solution described in this post uses the assumption of IAM roles to enforce MFA. STS can validate MFA information from devices that issue time-based one-time passwords (TOTPs).

A typical scenario involves the use of multiple AWS accounts (for example, Dev and Prod). One account is used for authentication and another contains the resource to be accessed (in this case, your AWS CodeCommit repository). You could also apply this solution to a single account.

This is what the workflow looks like:


  1. A user authenticates with IAM keys and a token from her MFA device and retrieves temporary credentials from STS. Temporary credentials consist of an access key ID, a secret access key, and a session token. The expiration of these keys can be configured with a duration of up to 36 hours.
  2. To access the resources in a different account, role delegation comes into play. The local Git repository is configured to use the temporary credentials to assume an IAM role that has access to the AWS CodeCommit repository. Here again, STS provides temporary credentials, but they are valid for a maximum of one hour.
  3. When Git is calling AWS CodeCommit, the credentials retrieved in step 2 are used to authenticate the requests. When the credentials expire, they are reissued with the credentials from step 1.

You could use permanent IAM keys to directly assume the role in step 2 without the temporary credentials from step 1. The process reduces the frequency with which a developer needs to use MFA by increasing the lifetime of the temporary credentials.

Account Setup Tasks

The tasks to set up the MFA scenario are as follows:

  1. Create a repository in AWS CodeCommit.
  2. Create a role that is used to access the repository.
  3. Create a group allowed to assume the role.
  4. Create a user with an MFA device who belongs to the group.

The following steps assume that you have set up the AWS CLI and configured it with the keys of users who have the required permissions to IAM and AWS CodeCommit in two accounts. Following the workflow, we will create the following admin users and AWS CLI profiles:

  • admin-account-a needs permissions to administer IAM (built-in policy IAMFullAccess)
  • admin-account-b needs permissions to administer IAM and AWS CodeCommit (built-in policies IAMFullAccess and AWSCodeCommitFullAccess)

At the time of this writing, AWS CodeCommit is available in us-east-1 only, so use that region for the region profile attribute for account B.

The following scripts work for Linux and Mac OS. For readability, line breaks are separated by back slashes. If you want to run these scripts on Microsoft Windows, you will need to adapt them or run them on an emulation layer (for example, Cygwin).

Replace placeholders like <XXXX> before issuing the commands.

Task 1: Create a repository in AWS CodeCommit

Create an AWS CodeCommit repository in Account B:

aws codecommit create-repository 
   --repository-name myRepository 
   --repository-description "My Repository" 
   --profile admin-account-b

Task 2: Create a role that is used to access the repository

  1. Create an IAM policy that grants access to the repository in Account B. Name it MyRepositoryContributorPolicy.

    Here is the MyRepositoryContributorPolicy.json policy document:

    "Version": "2012-10-17",
    "Statement": [
            "Effect": "Allow",
            "Action": [
            "Resource": [

    Create the policy:

    aws iam create-policy 
        --policy-name MyRepositoryContributorPolicy 
        --policy-document file://./MyRepositoryContributorPolicy.json 
        --profile admin-account-b

  2. Create a MyRepositoryContributorRole role that has the MyRepositoryContributorPolicy attached in Account B.

    Here is the MyRepositoryContributorTrustPolicy.json trust policy document:

      "Version": "2012-10-17",
      "Statement": [
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::<ACCOUNT_A_ID>:root"
          "Action": "sts:AssumeRole"

    Create the role:

    aws iam create-role 
    --role-name MyRepositoryContributorRole 
    --assume-role-policy-document file://./MyRepositoryContributorTrustPolicy.json 
    --profile admin-account-b

    Attach the MyRepositoryContributorPolicy:

    aws iam attach-role-policy 
    --role-name MyRepositoryContributorRole 
    --policy-arn arn:aws:iam::<ACCOUNT_B_ID>:policy/MyRepositoryContributorPolicy 
    --profile admin-account-b

Task 3: Create a group allowed to assume the role

  1. Create a MyRepositoryContributorAssumePolicy policy for users who are allowed to assume the role in Account A.

    Here is the MyRepositoryContributorAssumePolicy.json policy document:

        "Version": "2012-10-17",
        "Statement": [
                "Effect": "Allow",
                "Action": [
                "Resource": [
                "Condition": {
                    "NumericLessThan": {
                        "aws:MultiFactorAuthAge": "86400"

    The aws:MultiFactorAuthAge attribute is used to specify the validity, in seconds, after the temporary credentials with MFA information have been issued. After this period, the user can’t issue new temporary credentials by assuming a role. However, the old credentials retrieved by role assumption may still be valid for one hour to make calls to the repository.

    For this example, we set the value to 24 hours (86400 seconds).

    Create the policy:

    aws iam create-policy 
        --policy-name MyRepositoryContributorAssumePolicy 
        --policy-document file://./MyRepositoryContributorAssumePolicy.json 
        --profile admin-account-a

  2. Create the group for all users who need access to the repository:

    aws iam create-group 
        --group-name MyRepositoryContributorGroup 
        --profile admin-account-a

  3. Attach the policy to the group:

    aws iam attach-group-policy 
        --group-name MyRepositoryContributorGroup 
        --policy-arn arn:aws:iam::<ACCOUNT_A_ID>:policy/MyRepositoryContributorAssumePolicy 
        --profile admin-account-a

Task 4: Create a user with an MFA device who belongs to the group

  1. Create an IAM user in Account A:

    aws iam create-user 
        --user-name MyRepositoryUser 
        --profile admin-account-a

  2. Add the user to the IAM group:

    aws iam add-user-to-group 
        --group-name MyRepositoryContributorGroup 
        --user-name MyRepositoryUser 
        --profile admin-account-a

  3. Create a virtual MFA device for the user. You can use the AWS CLI, but in this case it is easier to create one in the AWS Management Console.

  4. Create IAM access keys for the user. Make note of the output of AccessKeyId and SecretAccessKey. They will be referenced as <ACCESS_KEY_ID> and <SECRET_ACCESS> later in this post.

    aws iam create-access-key 
       --user-name MyRepositoryUser 
       --profile admin-account-a

You’ve now completed the account setup. To create more users, repeat task 4. Now we can continue to the local setup of the contributor’s environment.

Initialize the Contributor’s Environment

Each contributor must perform the setup in order to have access to the repository.

Setup Tasks:

  1. Create a profile for the IAM user who fetches temporary credentials.
  2. Create a profile that is used to access the repository.
  3. Populate the role assuming profile with temporary credentials.

Task 1: Create a profile for the IAM user who fetches temporary credentials

By default, the AWS CLI maintains two files in ~/.aws/ that contain per-profile settings. One is credentials, which stores sensitive information for means of authentication (for example, the secret access keys). The other is config, which defines all other settings, such as the region or the MFA device to use.

Add the IAM keys for MyRepositoryUser that you created in Account Setup task 4 to ~/.aws/credentials:


Add the following lines to ~/.aws/config:

[profile FetchMfaCredentials]

get_session_token_duration_seconds is a custom attribute that is used later by a script. It must not exceed the value of aws:MultiFactorAuthAge that we used in the assume policy.

Task 2: Create a profile that is used to access the repository

Add the following lines to ~/.aws/config:

[profile MyRepositoryContributor]

When the MyRepositoryContributor profile is used, the MyRepositoryContributorRole is assumed with credentials of the MyRepositoryAssumer profile. You may have noticed that we have not put MyRepositoryAssumer in the credentials file yet. The following task shows how the file is populated.

Task 3: Populate the role assuming profile with temporary credentials

  1. Create the script in your home directory or any other location:

    # Parameter 1 is the name of the profile that is populated
    # with keys and tokens.
    # Parameter 2 is the name of the profile that calls the
    # session token service.
    # It must contain IAM keys and mfa_serial configuration
    # The STS response contains an expiration date/ time.
    # This is checked to only set the keys if they are expired.
    EXPIRATION=$(aws configure get expiration --profile "$1")
    if [ -n "$EXPIRATION" ];
            # get current time and expiry time in seconds since 1-1-1970
            NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
            # if tokens are set and have not expired yet
            if [[ "$EXPIRATION" > "$NOW" ]];
                    echo "Will not fetch new credentials. They expire at (UTC) $EXPIRATION"
    if [ "$RELOAD" = "true" ];
            echo "Need to fetch new STS credentials"
            MFA_SERIAL=$(aws configure get mfa_serial --profile "$2")
            DURATION=$(aws configure get get_session_token_duration_seconds --profile "$2")
            read -p "Token for MFA Device ($MFA_SERIAL): " TOKEN_CODE
            read -r AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN EXPIRATION AWS_ACCESS_KEY_ID < <(aws sts get-session-token 
                    --profile "$2" 
                    --output text 
                    --query 'Credentials.*' 
                    --serial-number $MFA_SERIAL 
                    --duration-seconds $DURATION 
                    --token-code $TOKEN_CODE)
            aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY" --profile "$KEY_PROFILE"
            aws configure set aws_session_token "$AWS_SESSION_TOKEN" --profile "$KEY_PROFILE"
            aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID" --profile "$KEY_PROFILE"
            aws configure set expiration "$EXPIRATION" --profile "$1"

    This script takes the credentials from the profile from the second parameter to request temporary credentials. These will be written to the profile specified in the first parameter.

  2. Run the script once. You might need to set execution permission (for example, chmod 755) before you run it.

    ~/ MyRepositoryAssumer FetchMfaCredentials
    Need to fetch new STS credentials
    Token for MFA Device (arn:aws:iam::<ACCOUNT_A_ID>:mfa/MyRepositoryUser): XXXXXX

    This populates information retrieved from STS to the ~/.aws/config and ~/.aws/credentials file.

  3. Clone the repository, configure Git to use temporary credentials, and create an alias to renew MFA credentials:

    git clone --config 'credential.helper=!aws codecommit 
        --profile MyRepositoryContributor 
        credential-helper $@' 
        --config 'credential.UseHttpPath=true' 
        --config 'alias.mfa=!~/ 
        MyRepositoryAssumer FetchMfaCredentials' 
        $(aws codecommit get-repository 
        --repository-name myRepository 
        --profile MyRepositoryContributor 
        --output text 
        --query repositoryMetadata.cloneUrlHttp)

    This clones the repository from AWS CodeCommit. You can issue subsequent calls as long as the temporary credentials retrieved in step 2 have not expired. As soon as they have expired, the credential helper will return an error with prompts for username and password:

    A client error (ExpiredToken) occurred when calling the AssumeRole operation:
    The security token included in the request is expired

    In this case, you should cancel the Git command (Ctrl-C) and trigger the renewal of the token by calling the alias in your repository:

    git mfa

We hope you find the steps for enforcing MFA for your repository users helpful. Feel free to leave your feedback in the comments.

Introducing Application Load Balancer – Unlocking and Optimizing Architectures

by George Huang | on | | Comments

This is a guest blog post by Felix Candelario & Benjamin F., AWS Solutions Architects.

This blog post will focus on architectures you can unlock with the recently launched Application Load Balancer and compare them with the implementations that use what we now refer to as the Classic Load Balancer. An Application Load Balancer operates at the application layer and makes routing and load-balancing decisions on application traffic using HTTP and HTTPS.

There are several features to help you unlock new workloads:

  • Content-based routing

    • Allows you to define rules that route traffic to different target groups based on the path of a URL. The target group typically represents a service in a customer’s architecture.
  • Container support

    • Provides the ability to load-balance across multiple ports on the same Amazon EC2 instance. This functionality specifically targets the use of containers and is integrated into Amazon ECS.
  • Application monitoring

    • Allows you to monitor and associate health checks per target group.

Service Segmentation Using Subdomains

Our customers often need to break big, monolithic applications into smaller service-oriented architectures while hosting this functionality under the same domain name.

In the architecture shown here, a customer has decided to segment services such as processing orders, serving images, and processing registrations. Each function represents a discrete collection of instances. Each collection of instances host several applications that provide a service.

Using a classic load balancer, the customer has to deploy several load balancers. Each load balancer points to the instances that represent and front the service by using a subdomain.

With the introduction of content-based routing on the new application load balancers, customers can reduce the number of load balancers required to accomplish the segmentation.

Application Load Balancers introduce the concept of rules, targets, and target groups. Rules determine how to route requests. Each rule specifies a target group, a condition, and a priority. An action is taken when the conditions on a rule are matched. Targets are endpoints that can be registered as a member of a target group. Target groups are used to route requests to registered targets as part of the action for a rule. Each target group specifies a protocol and target port. You can define health checks per target group and you can route to multiple target groups from each Application Load Balancer.

A new architecture shown here accomplishes with a single load balancer what previously required three. Here we’ve configured a single Application Load Balancer with three rules.

Let’s walk through the first rule in depth. To configure the Application Load Balancer to route traffic destined to, we must complete five tasks.

  1. Create the Application Load Balancer.
  2. Create a target group.
  3. Register targets with the target group.
  4. Create a listener with the default rule that forwards requests to the default target group.
  5. Create a listener that forwards requests to the previously created target group.

To create the Application Load Balancer, we must provide a name for it and a minimum of two subnets.

aws elbv2 create-load-balancer –name example-loadbalancer –subnets "subnet-9de127c4" "subnet-0b1afc20"

To create a target group, we must specify a name, protocol, port, and vpc-id. Based on the preceding figure, we execute the following command to create a target group for the instances that represent the order-processing functionality.

aws elbv2 create-target-group –name order-instances –protocol HTTP –port 80 –vpc vpc-85a268e0

After the target group has been created, we can either add instances manually or through the use of an Auto Scaling group. To add an Auto Scaling group, we use the Auto Scaling group name and the generated target group ARN:

aws autoscaling attach-load-balancer-target-groups –auto-scaling-group-name order_autoscaling_group –target-group-arns "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/order-instances/f249f89ef5899de1"

If we want to manually add instances, we would supply a list of instances and the generated target group ARN to register the instances associated with the order-processing functionality:

aws elbv2 register-targets –target-group-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/order-instances/f249f89ef5899de1" –targets Id=i-01cb16f914ec4714c,Port=80

After the instances have been registered with the target group, we create a listener with a default rule that forwards requests to the first target group. For the sake of this example, we’ll assume that the orders target group is the default group:

aws elb create-listener –load-balancer-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/order-instances/f249f89ef5899de1"  –protocol HTTP –port 80 –default-actions Type=forward,TargetGroupArn="arn:aws:elasticloadbalancing:us-east-1:007038732177:targetgroup/orders-instances/e53f8f9dfaf230c8"

Finally, we create a rule that forwards a request to the target group to which the order instances are registered when the condition of a path-pattern (in this case, ‘/orders/*’) is met:

aws elbv2 create-rule –listener-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:listener/app/example-loadbalancer/6bfa6ad4a2dd7925/6f916335439e2735" –conditions Field=path-pattern,Values=’/orders/*’ –priority 20 –actions Type=forward,TargetGroupArn="arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/order-instances/f249f89ef5899de1"

We repeat this process (with the exception of creating the default listener) for the images and registration functionality.

With this new architecture, we can move away from segmenting functionality based on subdomains and rely on paths. In this way, we preserve the use of a single subdomain, www, throughout the entire user experience. This approach reduces the number of Elastic Load Balancing load balancers required, which results in costs savings. It also reduces the operational overheard required for monitoring and maintaining additional elements in the application architecture.

Important The move from subdomain segmentation to path segmentation requires you to rewrite code to accommodate the new URLs.

Service Segmentation Using a Proxy Layer

A proxy layer pattern is used when customers want to use a single subdomain, such as www, while still segmenting functionality by grouping back-end servers. The following figure shows a common implementation of this pattern using the popular open source package NGINX.

In this implementation, the subdomain of is associated with a top-level external load balancer. This load balancer is configured so that traffic is distributed to a group of instances running NGINX. Each instance running NGINX is configured with rules that direct traffic to one of the three internal load balancers based on the path in the URL.

For example, when a user browses to, the external Elastic Load Balancing load balancer sends all traffic to the NGINX layer. All three of the NGINX installations are configured in the same way. When one of the NGINX instances receives the request, it parses the URL, matches a location for “/amazing”, and sends traffic to the server represented by the internal load balancer fronting the group of servers providing the Amazing Brand functionality.

It’s important to consider the impact of failed health checks. Should one of the NGINX instances fail health checks generated by the external load balancer, this load balancer will stop sending traffic to that newly marked unhealthy host. In this scenario, all of the discrete groups of functionality would be affected, making troubleshooting and maintenance more complex.

The following figure shows how customers can achieve segmentation while preserving a single subdomain without having to deploy a proxy layer.

In this implementation, both the proxy layer and the internal load balancers can be removed now that we can use the content-based routing associated with the new application load balancers. Using the previously demonstrated rules functionality, we can create three rules that point to different target groups based on different path conditions.

For this implementation, you’ll need to create the application load balancer, create a target group, register targets to the target group, create the listener, and create the rules.

1. Create the application load balancer.

aws elbv2 create-load-balancer –name example2-loadbalancer –subnets "subnet-fc02b18b" "subnet-63029106"

2. Create three target groups.

aws elbv2 create-target-group –name amazing-instances –protocol HTTP –port 80 –vpc vpc-85a268e0

aws elbv2 create-target-group –name stellar-instances –protocol HTTP –port 80 –vpc vpc-85a268e0

aws elbv2 create-target-group –name awesome-instances –protocol HTTP –port 80 –vpc vpc-85a268e0

3. Register targets with each target group.

aws elbv2 register-targets –target-group-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/amazing-instances/ad4a2174e7cc314c" –targets Id=i-072db711f70c36961,Port=80

aws elbv2 register-targets –target-group-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/stellar-instances/ef828b873624ba7a" –targets Id=i-08def6cbea7584481,Port=80

aws elbv2 register-targets –target-group-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/awesome-instances/116b2df4cd7fcc5c" –targets Id=i-0b9dba5b06321e6fe,Port=80

4. Create a listener with the default rule that forwards requests to the default target group.

aws elbv2 create-listener –load-balancer-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:loadbalancer/app/example2-loadbalancer/a685c68b17dfd091" –protocol HTTP –port 80 –default-actions Type=forward,TargetGroupArn="arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/amazing-instances/ad4a2174e7cc314c"

5.  Create a listener that forwards requests for each path to each target group. You need to make sure that every priority is unique.

aws elbv2 create-rule –listener-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:listener/app/example2-loadbalancer/a685c68b17dfd091/546af7daf3bd913e" –conditions Field=path-pattern,Values=’/amazingbrand/*’ –priority 20 –actions Type=forward,TargetGroupArn="arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/amazing-instances/ad4a2174e7cc314c"


aws elbv2 create-rule –listener-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:listener/app/example2-loadbalancer/a685c68b17dfd091/546af7daf3bd913e" –conditions Field=path-pattern,Values=’/stellarbrand/*’ –priority 40 –actions Type=forward,TargetGroupArn="arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/stellar-instances/ef828b873624ba7a"


aws elbv2 create-rule –listener-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:listener/app/example2-loadbalancer/a685c68b17dfd091/546af7daf3bd913e" –conditions Field=path-pattern,Values=’/awesomebrand/*’ –priority 60 –actions Type=forward,TargetGroupArn="arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/awesome-instances/116b2df4cd7fcc5c"

This implementation not only saves you the costs associated with running instances that support a proxy layer and an additional layer of load balancers. It also increases robustness as a result of application monitoring. In the Classic Load Balancer implementation of a proxy pattern, the failure of a single instance hosting NGINX impacts all of the other discrete functionality represented by the grouping of instances. In the application load balancer implementation, health checks are now associated with a single target group only. Failures and performance are now segmented from each other.

Run the following command to verify the health of the registered targets in the Amazing Brands target group:

aws elbv2 describe-target-health –target-group-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/amazing-instances/ad4a2174e7cc314c"

If the instances in this target group were marked as unhealthy, you would see the following output:


    "TargetHealthDescriptions": [


            "HealthCheckPort": "80",

            "Target": {

                "Id": "i-072db711f70c36961",

                "Port": 80


            "TargetHealth": {

                "State": "unhealthy",

                "Reason": "Target.Timeout",

                "Description": "Request timed out"





Service Segmentation Using Containers

Increasingly, customers are using containers as a way to package and isolate applications. Instead of grouping functionality by instances, customers are providing an even more granular collection of computing resources by using containers.

When you use Classic load balancers, you create a fixed relationship between the load balancer port and the container instance port. For example, it is possible to map the load balancer port 80 to the container instance port 3030 and the load balancer port 4040 to the container instance port 4040. However, it is not possible to map the load balancer port 80 to port 3030 on one container instance and port 4040 on another container instance.

The following figure illustrates this limitation. It also points out a pattern of using a proxy container to represent other containers operating on different ports. Logically, this implementation is similar to the proxy segmentation implementation described earlier.

Figure 5 Classic load balancer container based segmentation

Enhanced container support is one of the major features of the Application Load Balancer. It makes it possible to load-balance across multiple ports on the same EC2 instance. The following figure shows how this capability removes the need to run containers that proxy access to other containers.

To integrate containers, you only need to register the targets in the target group, which the Amazon ECS scheduler handles automatically. The following command configures /cart as illustrated in the preceding figure.

aws elbv2 register-targets –-target-group-arn "arn:aws:elasticloadbalancing:us-west-2:007038732177:targetgroup/cart-instances/ad4a2174e7cc314c" –-targets Id=i-84ri3a2c6dcd16b9c,Port=90 Id=i-83fc3a2c6dcd16b9c,Port=90 Id=i-qy342a2c6dcd16b9c,Port=100

A/B Testing

A/B testing is a term used for randomized experiments of two separate website experiences to test and gather data that will be helpful in decision-making. To facilitate this type of testing, you need to redirect a percentage of traffic to the secondary stack.

By using Classic Load Balancers, you can conduct these experiments by grouping the different experiences under separate load balancers. By using Amazon Route 53, you can then leverage a group of weighted resource record sets that point to the CNAMEs provided by the Classic Load Balancer. By modifying the weight of a given record, you can then move a random sampling of customers to a different website experience represented by the instances behind the Classic Load Balancer.

The introduction of the application load balancer optimizes A/B testing in a couple of ways. In the following figure, you can see the same grouping of instances that represent the two different website experiences (the A and the B experience shown in the preceding figure). The major differences here are one less load balancer, which reduces costs and configuration, and a new mechanism, rules, to control the switch from the A to the B experience. In this configuration, the logic for redirecting a percentage of traffic must be done at the application level, not the DNS level, by rewriting URLs that point to the B stack instead of the default A stack. The benefits of this approach are that specific users are targeted based on criteria that the application is aware of (random users, geographies, users’ history or preferences). There is also no need to rely on DNS for redirecting some traffic, so the control of who is directed to stack B is much more fine-grained. This mechanism also allows for a more immediate transitioning of users from the A to the B experience because there is no delay associated with DNS records having to be flushed for user cache.


The launch of the application load balancer provides significant optimization in segmentation techniques and A/B testing. These two use cases represent only a subset, but they illustrate how you can leverage the new features associated with this launch. Feel free to leave your feedback in the comments.

Building a Microsoft BackOffice Server Solution on AWS with AWS CloudFormation

by Bill Jacobi | on | in Best practices, How-to, New stuff | | Comments

Last month, AWS released the AWS Enterprise Accelerator: Microsoft Servers on the AWS Cloud along with a deployment guide and CloudFormation template. This blog post will explain how to deploy complex Windows workloads and how AWS CloudFormation solves the problems related to server dependencies.

This AWS Enterprise Accelerator solution deploys the four most requested Microsoft servers ─ SQL Server, Exchange Server, Lync Server, and SharePoint Server ─ in a highly available, multi-AZ architecture on AWS. It includes Active Directory Domain Services as the foundation. By following the steps in the solution, you can take advantage of the email, collaboration, communications, and directory features provided by these servers on the AWS IaaS platform.  

There are a number of dependencies between the servers in this solution, including:

  • Active Directory
  • Internet access
  • Dependencies within server clusters, such as needing to create the first server instance before adding additional servers to the cluster.
  • Dependencies on AWS infrastructure, such as sharing a common VPC, NAT gateway, Internet gateway, DNS, routes, and so on.

The infrastructure and servers are built in three logical layers. The Master template orchestrates the stack builds with one stack per Microsoft server and manages inter-stack dependencies. Each of the CloudFormation stacks uses PowerShell to stand up the Microsoft servers at the OS level. Before it configures the OS, CloudFormation configures the AWS infrastructure required by each Windows server. Together, CloudFormation and PowerShell create a quick, repeatable deployment pattern for the servers. The solution supports 10,000 users. Its modularity at both the infrastructure and application level enables larger user counts.

MSServers Solution - 6 CloudFormation Stacks

Managing Stack Dependencies

To explain how we enabled the dependencies between the stacks, the SQLStack is dependent on ADStack since SQL Server is dependent on Active Directory; and, similarly, SharePointStack is dependent on SQLStack, both as required by Microsoft. Lync is dependendent on Exchange since both servers must extend the AD schema independently. In Master, these server dependencies are coded in CloudFormation as follows:

"Resources": {
       "ADStack": …AWS::CloudFormation::Stack…
       "SQLStack": {
             "Type": "AWS::CloudFormation::Stack",
             "DependsOn": "ADStack",

             "Properties": …
"Resources": {
       "ADStack": …AWS::CloudFormation::Stack…
       "SQLStack": {
             "Type": "AWS::CloudFormation::Stack",
             "DependsOn": "ADStack",
             "Properties": …
       "SharePointStack": {
            "Type": "AWS::CloudFormation::Stack",
            "DependsOn": "SQLStack",
            "Properties": …

The “DependsOn” statements in the stack definitions forces the order of stack execution to match the diagram. Lower layers are executed and successfully completed before the upper layers. If you do not use “DependsOn”, CloudFormation will execute your stacks in parallel. An example of parallel execution is what happens after ADStack returns SUCCESS. The two higher-level stacks, SQLStack and ExchangeStack, are executed in parallel at the next level (layer 2).  SharePoint and Lync are executed in parallel at layer 3. The arrows in the diagram indicate stack dependencies.

Passing Parameters Between Stacks

If you have concerns about how to pass infrastructure parameters between the stack layers, let’s use an example in which we want to pass the same VPCCIDR to all of the stacks in the solution. VPCCIDR is defined as a parameter in Master as follows:

            "AllowedPattern": "[a-zA-Z0-9]+\..+",
            "Default": "",
            "Description": "CIDR Block for the VPC",
            "Type": "String"

By defining VPCCIDR in Master and soliciting user input for this value, this value is then passed to ADStack by the use of an identically named and typed parameter between Master and the stack being called.

            "Description": "CIDR Block for the VPC",
            "Type": "String",
            "Default": "",
            "AllowedPattern": "[a-zA-Z0-9]+\..+"

After Master defines VPCCIDR, ADStack can use “Ref”: “VPCCIDR” in any resource (such as the security group, DomainController1SG) that needs the VPC CIDR range of the first domain controller. Instead of passing commonly-named parameters between stacks, another option is to pass outputs from one stack as inputs to the next. For example, if you want to pass VPCID between two stacks, you could accomplish this as follows. Create an output like VPCID in the first stack:

Outputs”  : {
               “VPCID” : {
                          “Value” : “ {“Ref” : “VPC”},
                          “Description” : “VPC ID”
               }, …

In the second stack, create a parameter with the same name and type:

Parameters” : {
               “VPCID” : {
                          “Type” : “AWS::EC2::VPC::Id”,
               }, …

When the first template calls the second template, VPCID is passed as an output of the first template to become an input (parameter) to the second.

Managing Dependencies Between Resources Inside a Stack

All of the dependencies so far have been between stacks. Another type of dependency is one between resources within a stack. In the Microsoft servers case, an example of an intra-stack dependency is the need to create the first domain controller, DC1, before creating the second domain controller, DC2.

DC1, like many cluster servers, must be fully created first so that it can replicate common state (domain objects) to DC2.  In the case of the Microsoft servers in this solution, all of the servers require that a single server (such as DC1 or Exch1) must be fully created to define the cluster or farm configuration used on subsequent servers.

Here’s another intra-stack dependency example: The Microsoft servers must fully configure the Microsoft software on the Amazon EC2 instances before those instances can be used. So there is a dependency on software completion within the stack after successful creation of the instance, before the rest of stack execution (such as deploying subsequent servers) can continue. These intra-stack dependencies like “software is fully installed” are managed through the use of wait conditions. Wait conditions are CloudFormation resources just like EC2 instances and allow the “DependsOn” attribute mentioned earlier to manage dependencies inside a stack. For example, to pause the creation of DC2 until DC1 is complete, we configured the following “DependsOn” attribute using a wait condition. See (1) in the following diagram:

"DomainController1": {
            "Type": "AWS::EC2::Instance",
            "DependsOn": "NATGateway1",
            "Metadata": {
                "AWS::CloudFormation::Init": {
                    "configSets": {
                        "config": [
                    }, …
             "Properties" : …
"DomainController2": {
             "Type": "AWS::EC2::Instance",
[1]          "DependsOn": "DomainController1WaitCondition",
             "Metadata": …,
             "Properties" : …

The WaitCondition (2) uses on a CloudFormation resource called a WaitConditionHandle (3), which receives a SUCCESS or FAILURE signal from the creation of the first domain controller:

"DomainController1WaitCondition": {
            "Type": "AWS::CloudFormation::WaitCondition",
            "DependsOn": "DomainController1",
            "Properties": {
                "Handle": {
[2]                    "Ref": "DomainController1WaitHandle"
                "Timeout": "3600"
     "DomainController1WaitHandle": {
[3]            "Type": "AWS::CloudFormation::WaitConditionHandle"

SUCCESS is signaled in (4) by cfn-signal.exe –exit-code 0 during the “finalize” step of DC1, which enables CloudFormation to execute DC2 as an EC2 resource via the wait condition.

                "finalize": {
                       "commands": {
                           "a-signal-success": {
                               "command": {
                                   "Fn::Join": [
[4]                                            "cfn-signal.exe -e 0 "",
                                               "Ref": "DomainController1WaitHandle"


If the timeout had been reached in step (2), this would have automatically signaled a FAILURE and stopped stack execution of ADStack and the Master stack.

As we have seen in this blog post, you can create both nested stacks and nested dependencies and can pass parameters between stacks by passing standard parameters or by passing outputs. Inside a stack, you can configure resources that are dependent on other resources through the use of wait conditions and the cfn-signal infrastructure. The AWS Enterprise Accelerator solution uses both techniques to deploy multiple Microsoft servers in a single VPC for a Microsoft BackOffice solution on AWS.  

In a future blog post, we will illustrate how PowerShell can be used to bootstrap and configure Windows instances with downloaded cmdlets, all integrated into CloudFormation stacks.

AWS OpsWorks Endpoints Available in 11 Regions

by Daniel Huesch | on | | Comments

AWS OpsWorks, a service that helps you configure and operate applications of all shapes and sizes using Chef automation, has just added support for the Asia Pacific (Seoul) Region and launched public endpoints in Frankfurt, Ireland, N. California, Oregon, Sao Paolo, Singapore, Sydney, and Tokyo.

Previously, customers had to manage OpsWorks stacks for these regions using our N. Virginia endpoint. Using an OpsWorks endpoint in the same region as your stack reduces API latencies, improves instance response times, and limits impact from cross-region dependency failures.

A full list of endpoints can be found in AWS Regions and Endpoints.

Introducing the AWS for DevOps Getting Started Guide

by Paul Cornell | on | in How-to | | Comments

We are pleased to announce the AWS for DevOps Getting Started Guide is now available. As a companion to our DevOps and AWS website, this new resource teaches you, in a hands-on way, how to use services like AWS CodeCommit, AWS CodeDeploy, and AWS CodePipeline for continuous integration and continuous delivery.

Specifically, you will learn how to:

  1. Use AWS CloudFormation to give users access to required AWS services, resources, and actions.
  2. Create a source code repository in AWS CodeCommit and then use AWS CloudFormation to launch an Amazon EC2 instance that connects to the repository.
  3. Download the source code you will deploy and then push it into the repository.
  4. Use AWS CloudFormation to create the deployment target (an Amazon EC2 instance) and AWS resources that are compatible with AWS CodeDeploy, AWS Elastic Beanstalk, or AWS OpsWorks.
  5. Use AWS CloudFormation to create and run a pipeline in AWS CodePipeline to automate continuous delivery of the repository’s source code to the deployment target.
  6. Verify the deployment’s results on the deployment target.
  7. Make a change to the source code and then push it into the repository, triggering an automatic redeployment to the deployment target.
  8. Verify the deployed change on the deployment target.
  9. Use AWS CloudFormation to clean up the resources you created for this walkthrough.

You do not need to know anything about AWS to try this walkthrough. If you don’t have an AWS account, we’ll show you how to create one. And, if you’re new to AWS, our AWS Free Tier lets you to experiment with few, if any, charges to your AWS account.

Perhaps your organization is considering a move to DevOps practices. Or maybe your organization is practicing DevOps now. In either case, if you’re not yet using AWS services, this resource can help you down this path. Please let us know what you think by using the Feedback button on any of the walkthrough’s pages.

Get started now!