Field Notes: Using Infrastructure as Code to Manage Your AWS Networking Environment

Infrastructure as Code (IaC) brings automation to the provisioning process, which was traditionally done manually. Rather than relying on manually performed steps, both administrators and developers can instantiate infrastructure using configuration files. IaC helps avoid configuration drift through automation, and increases the speed and agility of infrastructure deployments. It also helps reduce errors and enhances the ability to apply changes through different stages consistently.

In this blog post, I demonstrate how to create a networking stack on AWS using AWS CloudFormation and manage it via IaC concepts and tools. In this blog, we create a pipeline using AWS CodeCommit and AWS CodePipeline services. Once we have created the pipeline and other required constructs to manage the infrastructure as code, we add an Amazon Managed Virtual Private Network (VPN) connection, Amazon VPC (VPC) Endpoints, VPC peering, and AWS Transit Gateway. We do this by modifying the code instead of making changes from the AWS web console. The resulting network architecture is shown in Figure 1.

Figure 1 – Network Architecture

It consists of three Amazon VPCs (named VPC1, VPC2 and VPC3) each with Internet Gateway and VPC endpoints for EC2 API and Amazon S3. VPC1 has an Amazon Managed S2S VPN connection to a customer gateway. VPC1 and VPC2 are peered with each other using VPC peering connection.

All three VPCs are attached to AWS Transit Gateway (TGW) with appropriate route table entries for inter-VPC communication. Details on CIDRs, subnets, BGP ASN etc. can be found in the VPC parameter files explained later in the post.

Prerequisites

Readers of this blog post should be familiar with the following AWS services:

AWS CloudFormation
AWS Networking and Content Delivery services like Amazon VPC (VPC), AWS Transit Gateway (TGW), AWS Virtual Private Network (VPN), VPC Peering and VPC Endpoints.
AWS Cloud9
AWS CodeCommit
AWS CodePipeline
Git Commands
Source Control concepts

For this walkthrough, you should have the following:

AWS account with sufficient limits. You will be using 6 Amazon VPCs with Internet Gateways and VPC Endpoints, 6 Amazon EC2 instances, 12 NAT Gateways, 18 EIPs. You can create a service limit increase case from your support console if necessary.
AWS CLI installed and configured on the workstation from where you are going to create AWS Cloud9 Integrated Development Environment (IDE).
A base VPC from where we launch and control the CICD Pipeline. This VPC will not be used for any other purpose. You can follow the instructions to create a separate base VPC while following this blog post. Please note down the subnet-id of one of the public subnets. We use this subnet-id to launch AWS Cloud9 IDE.
Credentials configured in AWS CLI should have the required IAM permissions to spin up and modify the resources mentioned in this post. I am using a user-id with AdministratorAccess.
S3 bucket to store AWS CodePipeline artifacts.
Make sure that you deploy the solution to N.Virginia (us-east-1) Region and your AWS CLI default Region is us-east-1. If us-east-1 is not the default Region, please set the Regions explicitly while running AWS CLI commands using –regionregion us-east-1 switch.

Overview

Create an AWS Cloud9 IDE, which will be used to perform all subsequent steps mentioned in this blog post.
Dowload and unzip the supporting files for this blog post in a subdirectory of the AWS Cloud9 IDE.
Using AWS CLI, create an AWS CodeCommit repository and upload base AWS CloudFormation template and parameter files.
Using AWS CLI, create an AWS CloudFormation stack to create IAM roles/ permission and SNS topics required for this blog post. Review the IAM permissions and tweak them according to your needs and policies.
Clone the AWS CodeCommit repository to our AWS Cloud9 IDE and start modifying our stack via AWS CodePipeline.

The following diagram represents how the pipeline is set up.

Figure 2- The IaC pipeline

Stage 1 – The name of this stage is CFNSource, which uses an AWS CodeCommit repository as source. Any changes to the branch ‘master’ will trigger the pipeline. AWS CloudFormation template ‘vpc-stack.yaml’ and parameter files ‘uat-vpc-params.json’ and ‘prod-vpc-params.json’ are in the ‘master’ branch of this repository. As we are using AWS CloudFormation templates, we do not need a build stage. AWS CloudFormation validation process takes care of that.

Stage 2 – The name of this stage is UAT, which will create a test stack for you to review and approve the changes before it is deployed in production. It has three actions:

‘CreateTestCFNStack’ of category Deploy, where it creates the stack based on the configuration specified in ‘vpc-stack.yaml’.
‘UATApproval’ of category Approval, where it sends notification to the administrator’s email address and then it waits for manual approval. The Administrator will have to check the deployment and do some manual tests like connectivity checks before approving the stack.
‘ProceedToProd’ of category Deploy, where the stack is deleted and the controls are handed over to next stage called ‘Production’.

Stage 3 – The name of this stage is Production where the changes are deployed in production. It has three actions:

‘CreateProdChangeset’ of category Deploy. In this action, based on the difference between existing stack and new configuration, a CloudFormation change set is created.
‘ProdApproval’ of category Approval, where it send notification to the administrator’s email address and it waits for manual approval. The Administrator will have to verify the CloudFormation change set before approving the stack.
‘DeployProduction’ of category Deploy, where the CloudFormation change set is deployed.

Walkthrough

This should take approximately 60 minutes to complete:

1. From your workstation with AWS CLI, create an AWS Cloud9 EC2 development environment by executing the following command. Replace value of --subnet-id with the subnet-id from prerequisites #2 and replace value of --owner-arn with arn for your IAM user-id.

aws cloud9 create-environment-ec2 --name IaC-Net-env --description "IDE for network IaC" --instance-type t2.micro --subnet-id subnet-05251bba34ad9b2bf --automatic-stop-time-minutes 60 --owner-arn arn:aws:iam::12345678901:user/vijay

2. Log into AWS Cloud9 instance created in step 1 by following the instructions.

3. Download and un-compress the required files to a folder of AWS Cloud9 IDE. I use the ~/environment folder.

$ cd ~/environment/
$ wget https://github.com/aws-samples/aws-networking-infra-via-iaac-principles/archive/master.zip 

$ unzip IaCBlog.zip 

$ ls -l
total 440
-rw-r--r-- 1 ec2-user ec2-user  21932 Jul 30 12:23 IaCBlog.zip
-rw-r--r-- 1 ec2-user ec2-user  15373 Jul 30 09:50 IaCBlogIAMRoles.yaml
-rw-r--r-- 1 ec2-user ec2-user   5699 Jul 29 13:07 IaCBlogPipeline.yaml
-rw-r--r-- 1 ec2-user ec2-user   1629 Jul 29 12:14 IaCBlogSNSTopics.yaml
-rw-r--r-- 1 ec2-user ec2-user  33359 Jul 29 20:26 add-Add-TGW-vpc-stack.yaml
-rw-r--r-- 1 ec2-user ec2-user  27992 Jul 29 19:23 add-Endpoints-vpc-stack.yaml
-rw-r--r-- 1 ec2-user ec2-user  29998 Jul 29 20:25 add-Peering-vpc-stack.yaml
-rw-r--r-- 1 ec2-user ec2-user    839 Jul 29 14:55 add-VPCs-prod-vpc-params.json
-rw-r--r-- 1 ec2-user ec2-user    840 Jul 29 14:55 add-VPCs-uat-vpc-params.json
-rw-r--r-- 1 ec2-user ec2-user  24732 Jul 29 12:14 add-VPCs-vpc-stack.yaml
-rw-r--r-- 1 ec2-user ec2-user    443 Jul 29 14:55 add-vpn-prod-vpc-params.json
-rw-r--r-- 1 ec2-user ec2-user    444 Jul 29 14:55 add-vpn-uat-vpc-params.json
-rw-r--r-- 1 ec2-user ec2-user   9888 Jul 29 12:14 add-vpn-vpc-stack.yaml
-rw-r--r-- 1 ec2-user ec2-user    337 Jul 29 12:14 prod-vpc-params.json
-rw-r--r-- 1 ec2-user ec2-user    336 Jul 29 12:14 uat-vpc-params.json
-rw-r--r-- 1 ec2-user ec2-user   7979 Jul 29 12:14 vpc-stack.yaml
$

4. From one of the terminal windows of AWS Cloud9 environment, generate and save keys for EC2 instances.

Create a Key Pair

aws ec2 create-key-pair \
--key-name MyKeyPairIaC \
--query 'KeyMaterial' \
--output text > MyKeyPairIaC.pem

Change permission of the private key pair file.

chmod 400 MyKeyPairIaC.pem

5. Create an AWS CodeCommit repository and note the clone URLs “cloneUrlHttp” and “cloneUrlSsh”. The following command will create a AWS CodeCommit repository named ‘IaCBlogRepo’.

aws codecommit create-repository --repository-name IaCBlogRepo --repository-description "IaC Repository"

Output:

{

<snip>

"cloneUrlHttp": "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/IaCBlogRepo",

"cloneUrlSsh": "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/IaCBlogRepo",

<snip>

}

6. Clone the AWS CodeCommit repository to your AWS Cloud9 IDE by running the below git command with“cloneUrlHttp”copied in step 5.

git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/IaCBlogRepo

This will create an empty folder by the repository name. All the git commands mentioned in this post will have to be executed from this directory.

7. Copy vpc-stack.yml, uat-vpc-prams.json and prod-vpc-params.json to the newly created directory and push it to the repository.

Note: You can make changes to uat-vpc-params.json and prod-vpc-params.json to change the VPC CIDR, Subnets, Tag prefix, Bastion host AMI id or bastion host ssh key by editing it in your AWS Cloud9 IDE. If you chose to do so, then make sure that you replicate those changes to add-vpn-uat-vpc-params.json, add-vpn-prod-vpc-params.json, add-VPCs-uat-vpc-params.json and add-VPCs-prod-vpc-params.json parameter files as well before proceeding.

$ cd IaCBlogRepo
$ cp ../vpc-stack.yaml .
$ cp ../uat-vpc-params.json .
$ cp ../prod-vpc-params.json .
$ git add .; git commit -m "Added base files"; git push origin master

8. Create IAM roles to be used by AWS Code Pipeline and AWS CloudFormation using AWS CloudFormation template IaCBlogIAMRoles.yaml downloaded in the beginning.

$ cd IaCBlogRepo
$ cp ../vpc-stack.yaml .
$ cp ../uat-vpc-params.json .
$ cp ../prod-vpc-params.json .
$ git add .; git commit -m "Added base files"; git push origin master

Check the status of the stack by running the below command every few minutes. You should see StackStatus value as CREATE_COMPLETE.

aws cloudformation describe-stacks --stack-name BlogIaCIAMRoles | grep StackStatus

9. Create Amazon SNS Topics to be used by AWS CodePipeline to send notifications using template IaCBlogSNSTopics.yaml downloaded in the beginning. Replace example@exmaple.com with the email address you want to use to get notification for approvals.

aws cloudformation create-stack \
--stack-name BlogIaCSNSTopics \
--template-body file://IaCBlogSNSTopics.yaml \
--parameters ParameterKey=ProdApprovalEmail,ParameterValue=example@example.com ParameterKey=ProdTopic,ParameterValue=IaCBlogProd ParameterKey=UATApprovalEmail,ParameterValue=example@example.com ParameterKey=UATTopic,ParameterValue=IaCBlogUAT \
--capabilities CAPABILITY_NAMED_IAM

Check the status of the stack by executing the below command every few minutes. You should see StackStatus value as CREATE_COMPLETE.

aws cloudformation describe-stacks --stack-name BlogIaCSNSTopics | grep StackStatus

Once the stack is created, you will receive subscription confirmation emails on the email addresses used in above command. Click on the ‘Confirm Subscription’ link in the email before proceeding to the next step.

10. Create the AWS CodePipeline to manage networking infrastructure as code. Replace the S3 bucket in below command with one in your account or with one which is accessible from your account. I am using a bucket in my account named ‘blogiacbucket’.

aws cloudformation create-stack \
--stack-name BlogIaCPipeLine \
--template-body file://IaCBlogPipeline.yaml \
--parameters ParameterKey=CFNTemplateRepoName,ParameterValue=IaCBlogRepo ParameterKey=ArtifactStoreS3Location,ParameterValue=blogiacbucket ParameterKey=IAMStackName,ParameterValue=BlogIaCIAMRoles ParameterKey=SNSStackName,ParameterValue=BlogIaCSNSTopics \
--capabilities CAPABILITY_NAMED_IAM

Check the status of the stack by executing the below command every few minutes. You should see StackStatus value as CREATE_COMPLETE.

aws cloudformation describe-stacks --stack-name BlogIaCPipeLine | grep StackStatus

11. Review the pipeline: Open the CodePipeline from AWS console page. On the All Pipelines page, choose the name of the pipeline. You will see the three stages explained earlier in the blog post. At this point the pipeline will already be executing.

Once the test stack is up, pipeline will send an email notification to the email addresses used in step 9 for approval. Before approving you can verify the VPC and other resources like NAT Gateway, routing table etc.
Once verified, you can click on the link in email or click on Review under UATApproval action in AWS CodePipeline and then click on Approve after entering comments. Once approved the 3rd action will go ahead and delete the test stack and proceed with deploying the Prod Stack.

12. Once the UAT stage is complete, the pipeline will create a ChangeSet for Prod Stack and send you an approval notification email.

To verify the changes, from the AWS console, go to Cloudformation, then select the recently launched Stack for which the status is “REVIEW_IN_PROGRESS”.
Click on Change Sets tab and then on the Change Set name to review the changes to current stack.

13. Once verified you can approve the changes by clicking on Review and then Approve after entering approval comment in CodePipeline screen. At the end of this action, the pipeline will deploy the changes to the production stack.

Making changes to the stack by modifying the CloudFormation template

Adding VPN to the first VPC (VPC1):

Copy files add-vpn-vpc-stack.yaml, add-vpn-uat-vpc-params.json and add-vpn-prod-vpc-params.json to the directory where the AWS CodeCommit repository was cloned, as vpc-stack.yaml, uat-vpc-params.json and prod-vpc-params.json respectively.
If you examine the content of these new files, you will see that it appends the VPN configuration and parameters to the current template.
You can change remote VPN device, CIDR and BGP ASN by modifying uat-vpc-params.json and prod-vpc-params.json files.

$ cd IaCBlogRepo
$ cp ../add-vpn-vpc-stack.yaml ./vpc-stack.yaml
$ cp ../add-vpn-uat-vpc-params.json ./uat-vpc-params.json
$ cp ../add-vpn-prod-vpc-params.json ./prod-vpc-params.json

Before you push the changes to the repository, you can view the difference by running ‘git diff HEAD’ command. This will show the additional Resources and Parameters we added for the VPN. Once verified, you can push the changes to the repository by executing the following commands.

$ git add .; git commit -m "Added VPN to VPC1"; git push origin master